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ABSTRACT 


J  This  thesis  reviewed  45  studies  that  purported  to 
empirically  assess  the  effectiveness  of  information  systems. 
They  were  evaluated  for  (a)  vulnerability  to  alternative 
explanations  of  reported  findings,  (b)  effect  of  data 
collection  methods  upon  their  conclusions,  (c)  capacity  to 
establish  causal  inference,  (d)  how  they  defined  system 
success  and  (e)  reported  changes  in  organizations  following 
information  system  implementation. 

MIS  research  methods  do  not  provide  a  basis  for 
establishing  causal  inference.  This,  in  turn,  denies  the 
possibility  of  constructing  a  formal  theory  to  describe, 
explain  or  predict  the  successful  implementation  of 
information  systems.  This  deficiency  is  worsened  by  the 
emergence  of  antagonistic  schools  of  thought  among  MIS 
researchers  as  to  how  system  effectiveness  should  be 
evaluated. 

Recommendations  are  made  for  improvements  in  MIS  research 
designs  and  data  collection  techniques.  Suggestions  are  made 
for  ways  by  which  line  managers  and  commanders  as  well  as 
managers  of  MIS  activities  must  become  informed  consumers  of 
MIS  academic  research  on  system  effectiveness. 
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I .  INTRODUCTION 


The  point  of  implementing  computer-based  information 
systems  i3  to  gain  either  increased  operational  efficiency  or 
enhanced  managerial  effectiveness.  These  gains,  however,  are 
more  often  asserted  or  assumed  than  they  are  demonstrated  or 
tested  (Lotus,  1988;  Fawcette,  1988;  Bulkeley,  1987)  . 

Starting  in  1985,  senior  line  managers  and  information 
system  managers  in  the  United  States  began  to  ask  whether 
they  were  getting  a  payoff  from  their  investment  in 
information  systems,  especially  since  the  advent  of  the 
personal  computer. 

A  similar  skepticism  could  well  be  evident  in  other 
countries  (Maginnis,  1986;Bowen,  1986;  Strassmann,  1985; 
Wessel,  1988;  LaPlante,  1988) . 

A.  GROWING  DOUBT  ABOUT  THE  PAYOFF  FROM  INFORMATION  SYSTEMS 

Steve  Crummey  (1988),  of  Lotus  Development  Corporation, 
assessed  the  current  information  system  productivity  dilemma, 
at  least  as  it  applied  to  the  proliferation  of  desk  top 
microcomputers:  "In  1976,  at  the  dawn  of  the  personal 
computer  age,  corporations  invested  $100  million  in  PC 
(personal  computer)  technology.  Ten  years  later  in  1986,  they 
spent  $46  billion  on  PCs--and  nobody  really  knows  what  those 
dollars  arc  buying." 
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1 .  Questions  That  Axis* 

Seeds  of  doubt  about  the  assumption  that  the 
automation  of  management  information  systems  (MIS)  leads  to 
increased  productivity  provoke  questions  about  the  ways  in 
which  the  success  of  these  systems  are  evaluated: 

a.  What  is  meant  by  a  payoff,  success  or  benefit  from  the 
implementation  of  an  information  system? 

b.  What  are  the  magnitudes  of  improvement  in  organizational 
functioning  reported  by  formal  studies  of  information 
system  implementation? 

c.  What  research  tools  are  employed  to  gather  data  on  the 
success  or  failure  of  information  systems? 

d.  To  what  extent  do  the  findings  reported  by  formal  studies 
of  system  effectiveness  have  scientific  credibility? 

2.  Answering  Those  Questions 

To  address  these  questions,  this  thesis  reviewed  45 
formal  studies,  ostensibly  dealing  with  the  measurement  of 
the  effectiveness  of  information  systems.  They  were  examined 
for : 

a.  How  they  defined  effectiveness,  benefit  or  success  from 
a  system  implementation. 

b.  How  their  research  designs  dealt  with  threats  to 
validity . 

c.  The  means  by  which  they  gathered  data  on  system 
effectiveness . 

d.  Reported  levels  of  improvement  in  organizational 
functioning  following  the  implementation  of  a  system. 

The  findings  of  this  review  are  analyzed  for  their 
implications  for: 

Senior  line  managers  and  commanders  as  well  as  managers 
of  information  system  function. 

The  intellectual  foundations  of  the  study  of  management 
information  systems. 
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B.  THE  HEED  TO  STUDY  PAYOFFS  FROM  INFORMATION  SYSTEMS 

After  spending  billions  of  dollars  on  automating  the 
offices  of  white  collar  workers,  organizations  find  that  they 
must  now  scrutinize  the  claims  by  computer  vendors  about 
increased  productivity.  To  effectively  manage  a  resource  an 
organization  must  have  a  means  to  measure  its  benefits  as  well 
as  its  costs. 

An  appropriate  managerial  question  that  arises  is  whether 
an  organization  is  getting  a  return  on  its  investment  in  a 
resource.  This  necessity  is  not  diminished  because  the 
resource  in  question  is  a  management  information  system. 
Managers  of  information  systems  are  pressed  by  senior  line 
management  to  answer,  again  and  again,  the  question,  "What  is 
the  value  of  the  company's  investment  in  information  systems?" 
(Matlin,  1979) . 

According  to  Strassmann  (1976),  "Industries  and 
governments  all  over  the  world  are  currently  struggling  to 
contain  rising  administrative  and  clerical  overheads  by 
automating  information  handling  in  the  office." 

While  the  clerical  work  force  has  been  the  fastest  growing 
segment  of  the  labor  force  in  nearly  every  industrialized 
country  since  the  1960s,  the  productivity  of  this  sector  has 
been  flat  since  the  early  1970s  (Harris,  1987)  .  This  has  been 
the  result  despite  the  fact  that  40%  of  capital  spending  (in 
the  U.S.)  has  been  directed  at  information  technology. 

General  managers  in  the  United  States  surveyed  by  the 
University  of  Minnesota  Management  information  Systems 
Research  Center  (Brancheau  and  Wetherbe,  1987)  ranked  the 
measuring  of  system  effectiveness  as  the  fourth  most  critical 
problem  they  face  in  managing  information  systems.  Sixty  per 
cent  of  the  Management  Information  Systems  executives  polled 
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in  the  same  survey  placed  the  measuring  of  effectiveness  in 
their  Top  Ten  list  of  critical  problems.  The  assessment  of 
system  success  has  been  in  the  Top  Ten  list  of  critical  MIS 
problems  since  it  was  first  formulated  in  1980. 

C.  THE  NEED  TO  DEVELOP  A  FORMAL  THEORY  OF  SYSTEM 
EFFECTIVENESS 

Clearly,  then,  the  evaluation  of  the  success  of 
information  systems  is  viewed  by  senior  line  managers,  and  by 
MIS  managers,  as  being  in  the  forefront  of  their  concerns. 
While  these  executives  represent  the  sectors  who  have  the  most 
at  stake  in  the  successful  implementation  and  exploitation  of 
information  systems,  in  the  social  division  of  labor  they  are 
not  the  agents  of  measurement. 

That  task  falls  to  a  neutral  sector  of  the  information 
systems  realm:  the  academic  scholars  of  MIS.  Management  must 
examine  not  only  the  effectiveness  of  information  systems  but 
also  the  effectiveness  of  the  ways  by  which  effectiveness 
itself  is  studied.  It  is  the  job  of  the  MIS  scholar  to 
discover  and  systematize  what  is  known  about  system 
effectiveness.  A  theory  of  system  implementation  would  express 
what  is  known,  what  is  predicted  and  what  is  assumed  about  the 
success  of  information  systems.  Without  a  formal  theory  of 
system  implementation,  we  cannot  competently  describe,  explain 
or  predict. 

1 .  Functions  of  a  Theory  of  Systam  Effectiveness 

A  formal  theory  of  system  effectiveness  would  be  the 
basis  for  describing,  explaining  and  predicting  system 
effectiveness  (Denzin,  1970) : 

•  Descript ion- -Formal  theory  permits  the  description  of 
a  phenomenon  is  such  a  way  that  others  can  repeat  a 


4 


study  with  a  high  degree  of  agreement.  Formal  theory 
can  provide  the  descriptive  data  that  can  answer  the 
managerial  question  "What  is  the  payoff  of  information 
systems?" 


•  Explanation — Formal  theory  consists  of  a  set  of 
interrelated  propositions  such  that  some  can  be  deduced 
from  others  and,  together,  they  make  sense  out  of  what 
is  observed  (Homans,  1964) .  To  say  that  a  phenomenon 
has  been  explained  is  to  argue  that  its  occurrence  in 
the  future  can  be  predicted.  Explanation  assumes  that 
the  causality  problem  has  been  resolved:  that 

covariance,  time  order  and  the  exclusion  of  rival  causal 
factors  have  been  established. 


•  Predict ion--P rediction  is  the  test  of  explanation.  If 
we  claim  to  have  explained  why  a  given  set  of  variables 
occurs  together  we  must  be  able,  by  definition,  to 
predict  their  future  relationships. 


2.  What  a  Formal  Theory  of  System  Implementation  la  Mot 

A  formal  theory  of  system  implementation  is  something 
apart  from  what  is  usually  described  colloquially  by  the  term 
"theory . " 

a.  A  model  is  not  a  theory.  Models  are  sets  of  relationships 
built  upon  assumptions.  Some  of  these  assumptions  may  be 
based  upon  rigorous  empirical  assessment  or  upon  an 
empirical  base  of  everyday  experience.  But  a  model  itself 
is  a  metaphor;  it  is  used  as  an  aid  to  description. 
However,  a  model  is  not  a  foundation  for  deriving 
testable  hypotheses  from  interrelated  propositions. 

b.  Conceptual  frameworks  are  not  theories. 

c.  A  single  proposition,  say,  if  X  then  Y  is,  by  itself, 
not  a  formal  theory. 

d.  Nor  do  sets  of  propositions,  by  themselves,  constitute 
theory.  They  must  be  arranged  in  a  deductive  scheme 
(Homans,  1964 )  . 

e.  Taxonomies  are  not  theories  although  they  may  be  a 
starting  point  in  the  development  of  propositions  that 
can  lead  to  formal  theory. 
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f.  Criticism,  journalistic  observation  or  hunches  are  not 
theories . 

D.  ELEMENTS  OF  X  FORMAL  THEORY  OF  SYSTEM  IMPLEMENTATION 

1.  Formal  Thaory  as  Intarralatad  Propositions 

A  formal  theory  consists  of  a  set  of  testable 
propositions  that  are  related  in  such  a  way  that  some  are 
deduced  from  others.  That  is,  for  example,  Proposition  Three 
may  be  deduced  from  Propositions  One  and  Two  while  Proposition 
Four  can  be  deduced  from  Propositions  Two  and  Three. 

2 .  Tha  Function  of  Propositions  in  Building  Formal  Thaory 

The  propositions  that  are  the  first  level  of  analysis 
in  theory  construction  are  themselves  statements  of 
relationships  between  the  next  level  of  analysis,  the  concept. 
Concepts  are  the  bricks  in  the  building  of  formal  theory  held 
together  in  propositional  statements. 

In  the  proposition  If  X  then  Y  X  and  Y  are  the 
concepts.  The  use  of  concepts  is  critical  in  the  development 
of  a  formal  theory.  A  formal  theory,  like  the  proverbial 
chain,  is  only  as  strong  as  its  weakest  concept.  Concepts 
represent  pieces  of  reality.  They  have  three  functions: 

a.  Introduce  new  points  of  view  into  a  scientific  process. 

b.  Convert  perceptions  of  an  environment  into  scientific 
activity . 

c.  Make  possible  deductive  reasoning. 

3.  Concept*  as  tha  Basis  of  Propositions 

Through  concepts,  everyday  experience  can  be  raised 
to  the  level  of  shared  knowledge.  Nonetheless,  a  specific 
concept  is  a  commitment  to  a  single  viewpoint,  excluding  all 
others  by  its  formulation.  A  concept  is  also  a  plan  of  action 
to  observe  a  phenomenon.  Concepts  suggest  operational 
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activity  and  measurement.  At  the  conceptual  level,  as  shown 
in  Figure  1,  a  propositional  statement  takes  the  form  if  A, 
then  B.  If  user  irvolvement  is  high,  then  a  system 
implementation  will  be  successful. 

This  statement  is  put  into  a  testable  form  at  the 
operational  level.  It  is  expressed  in  the  form  that  a  high 
score  on  an  attitudinal  scale  indicating  user  involvement 
leads  to  a  high  score  on  an  attitudinal  scale  indicating  user 
satisfaction.  A  researcher  proposes  a  null  hypothesis  that  no 
relationship  exists  between  the  scales.  Specifically,  that, 
say,  a  Pearson  correlation  of  the  reported  scores  will  be 
zero . 

The  alternative  hypothesis  will  be  that  some 
relationship  exists  between  the  scales.  Again,  the  specific 
practice  will  be  to  reject  the  null  hypothesis  and  accept  the 
alternative  if  the  probability,  of  a  reported  Pearson 
correlation  (or  similar  gauge  of  the  association  of  two 
variables)  having  occurred  by  chance  alone,  is  less  than  .05. 


Conceptual  level 


Operational  level 


Concept  A 

Concept  B 

User 

Involvement 

Successful 

Implementation 

Indicator  A 

Indicator  B 

Attitudinal 

Attitudinal 

Questionnaire 

Questionnaire 

Scale  of 

Scale  of 

Involvement 

Satisfaction 

Figure  1.  Operationalizing  Concepts 
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a.  Definition  of  Concept s 

Defining  a  concept  is  a  critical  link  in  the 
conceptualization  of  theory.  A  definition  must  specify  what 
is  included  in  the  concept  (not  what  is  excluded)  .  They  should 
be  expressed  in  scientific  language,  not  in  everyday, 
colloquial  language. 

That  means  that  the  expression  should  have  a  high 
degree  of  consensus  among  its  users  as  to  what  it  refers. 
Moreover,  a  definition  must  not  be  a  tautology.  Rather,  it 
must  have  an  external  reference. 

b.  Nominal  Definitions 

This  is  the  primitive  level  of  defining  a  concept. 
Generally,  it  means  nothing  more  than  substituting  one  word 
for  another  or  employing  pseudo-mathematical  notation.  An 
example  would  be  the  use  of  a  capital  letter  S  to  be  a  symbol 
for  the  concept  of  self. 

Nominal  definitions  have  no  meaning  beyond  that 
arbitrarily  given  to  them  (Bierstadt,  1959) .  No  claim  to 
empirical  verification  is  possible.  A  nominal  definition 
cannot  be  proven  true  or  false  because,  by  definition,  it  is 
just  as  it  is  defined  (Denzin,  1970)  .  Being  true  by 
definition,  nominal  definitions  cannot  provide  a  basis  for 
causal  inference. 

At  root,  they  are  useless  for  the  construction  of 
propositions  for  formal  theory.  Nonetheless,  nominal 
definitions  are  not  without  utility.  The  use  of  pseudo- 
mathematical  notation  at  least  has  the  virtue  of  economizing 
on  space  in  published  matter. 

Further,  a  nominal  substitution  of  one  term  for 
another  that  is  burdened  with  emotional  baggage  allows  a 
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concept  to  be  treated  in  a  manner  that  is  more  scientific  and 
less  ideological.  Finally,  nominal  definitions  can  be  a 
medium  by  which  new  terms  and  concepts  are  introduced  into 
scientific  usage. 

c.  Real  Definition* 

A  real  definition  resolves  a  concept  into  its 
constituent  parts  (Denzin,  1970) .  For  example,  user 
involvement  can  be  defined  as  the  users  of  a  system  that 
participated  in  its  development  and  implementation.  This 
definition  can  be  looked  at  in  terms  of  users,  system, 
participation,  development  and  implementation.  It  points 
immediately  to  specific  steps  for  observation. 

The  attitudinal  scale  used  in  an  example  earlier 
would  need  to  have  sets  of  items  that  measured  the  degree  to 
which  a  respondent  was  (1)  a  user,  (2)  used  the  system  being 
evaluated,  (3)  participated  in  development  and  (4) 
participated  in  implementation. 

Moreover,  a  set  of  items  would  also  be  needed  to 
establish  the  meaning  of  the  term  "participation"  to  each 
respondent.  The  test  of  a  real  definition  is  made  in  terms 
of  the  empirical  observations  it  can  generate. 

d.  Operational  Definitions 

An  operational  definition  prescribes  what  to  do  to 
observe  a  concept.  It  is  the  vehicle  by  which  real  definitions 
are  translated  into  empirical  tools.  It  is  tantamount  to  an 
empirical  indicator  at  the  operational  level  of  a  proposition. 
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X.  CONDITIONS  TOR  ESTABLISHING  CAUSAL  INFERENCE 


1 .  Th«  Function  of  Causal  Infaranca  in  Building  Formal 

Tbaory 

The  foundation  of  formal  theory  is  a  set  of  testable 
propositions  that,  finally,  can  lead  from  explanations  to 
predictions.  That  implies  that  concepts  in  a  proposition 
must  be  expressed  in  causal  order.  The  proposition,  if 
Concept  X,  then  Concept  Y  requires  that  Concept  X  be  the  cause 
of  Concept  Y. 

2.  Three  Conditions  for  Establishing  Causality 

For  Concept  X  to  be  established  as  the  cause  of 
Concept  Y,  three  conditions  must  be  met: 

a.  Covariance:  both  X  and  Y  must  vary. 

b.  Time  order:  X  must  vary  first  followed  by  variance  in  Y. 

c.  Rival  hypotheses  that  can  explain  any  observed  variance 
in  Y  must  be  excluded. 

a .  Timm  Order 

Establishing  time  order  between  two  variables 
requires  that  the  design  of  an  empirical  investigation  must 
account  for  time.  Evidence  must  be  gathered  at  more  than  a 
single  point  in  time.  This  suggests  a  longitudinal  or  time- 
series  design.  If  the  effect  of  an  intervening  or  independent 
variable  X  upon  a  dependent  variable  Y  is  of  interest,  then 
the  design  must  involve  collection  of  data  before  the 
appearance  of  X  (observation  one:  01)  and  again  following  the 
intervention  (observation  two:  02)  .  A  pretest  observation  and 
a  posttest  observation  around  the  occurrence  of  the 
intervening  variable  X  is  expressed  in  this  notation: 

01  X  02 
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A  difference  between  01  and  02,  ceteris  paribus  may  be 
attributed  to  the  intervening  effect  of  X. 

b.  Rival  Explanations  tor  Obamrvmd  Changma 

Things,  of  course,  are  not  otherwise  equal. 
Alternative  explanations  arise  to  account  for  differences 
between  01  and  02.  These  rival  explanations  are  represented 
by  the  threats  to  internal  validity  (Campbell  and  Stanley, 
1966)  and  by  the  particular  techniques  by  which  data  are 
collected  (Denzin,  1970;  Phillips,  1971). 

Internal  validity  deals  with  the  issue  as  to 
whether  reported  findings  are  due  to  the  effect  of  an 
experimental  variable  or  due  to  other  causal  factors  for  which 
no  accounting  was  made  in  the  design  of  the  research.  Some 
threats  to  internal  validity  deal  with  spurious  conclusions 
based  upon  observations  made  before  and  after  the  effect  of 
an  intervening  experimental  variable.  Others  deal  with 
erroneous  conclusions  based  upon  supposed  differences  between 
comparison  groups . 

The  threats  to  internal  validity  are  (Campbell  and 
Stanley,  1966) : 

-  History  effects. 

-  Subject  maturation. 

-  Re-test  sophistication. 

-  Changes  in  instrument  calibration. 

-  Statistical  regression. 

-  Bias  in  selection  of  subjects. 

-  Differentia)  subject  drop-out  effect. 

-  Interaction  of  selection  bias  and  maturation  effect. 

The  content  and  affect  of  each  of  these  threats  are  explained 
in  detail  in  Chapter  III. 
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The  techniques  by  which  a  researcher  collects  data 
can  have  a  similar  effect  on  his  or  her  conclusions.  A  study 
might  report  an  observed  level  of  user  satisfaction  with  the 
implementation  of  a  new  system,  ascribing  this  finding  to  the 
effect  of  experimental  variable  X,  say  a  program  to  involve 
users  in  the  system's  design.  Yet,  in  reality,  the  observed 
difference  might  be  caused  by  the  very  wording  of  the 
questions  in  a  questionnaire  used  to  measure  satisfaction.  The 
order  in  which  the  questions  appear  in  a  questionnaire  or  the 
graphic  format  of  the  questionnaire  itself  could  also  create 
the  observed  effect. 

The  difficulty  with  the  ways  in  which  researchers 
collect  data  is,  according  to  Phillips  (1971)  that,  "...our 
measurements  of  most  variables  are  caused  less  by  the  'true' 
values  of  those  variables  than  by  the  effects  of  measuring 
them."  If  the  three  conditions  of  covariance,  time  order  and 
the  exclusion  of  rival  explanations  are  met,  then  a  researcher 
can  confirm  that  a  Concept  X  is  the  cause  of  a  Concept  Y.  If 
the  study  of  system  effectiveness  can  establish  causal 
inference  by  generating  causal  propositions,  then  it  has  the 
foundation  for  constructing  a  formal  theory  of  system 
effectiveness . 

If  students  of  information  systems  can  construct 
a  formal  theory  of  system  effectiveness,  then  the  discipline 
of  MIS  will  be  able  to  describe,  explain  and  predict  the 
conditions  under  which  an  implementation  of  an  information 
system  is  likely  to  be  a  success  or  a  failure.  If  an  academic 
discipline  is  not  engaged  in  the  construction  of  formal 
theories  that  describe,  explain  and  predict,  what  then  is  it 
doing.  If  a  field  of  inquiry  is  not  engaged  in  doing  science, 
it  risks  being  condemned  to  impotency  and  irrelevancy. 
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II.  LITERATURE  REVIEW 


A.  COST-BENEFIT  ANALYSIS 

Findings  on  whether  or  not  Cost-Benefit  Analysis  should 
be  used,  the  problems  and  weaknesses  associated  with  using  it 
and  the  methods  by  which  it  should  be  conducted  are  summarized 
below. 

Ives,  et.al  (1983),  states  that: 

Theoretically,  the  determination  of  information  system 
value  is  a  matter  of  economics:  the  costs  of  system 
operation  and  development  are  subtracted  from  the  actual 
benefits  (in  organizational  effectiveness)  to  obtain  the 
net  value  of  the  system  to  the  organization. 

In  practice:  1).  intangible  costs  and  especially  benefits 
of  the  information  systems  are  difficult  to  recognize  and 
to  convert  to  their  monetary  equivalent;  2) .  Some  DSS  are 
used  for  disparate,  relatively  unstructured,  ad  hoc 
decisions;  objectively  assessing  the  benefits  of  such 
systems  may  be  nearly  impossible;  3)  .  Data  on  system 
success  may  be  determinable  but  not  recorded  by  the 
organization  and,  therefore,  unavailable  for  research 
purposes . 


Keen  (March  1981),  suggest  that  traditional  cost-benefit 
analysis  is  not  well-suited  to  Decision  Support  System. 
However,  he  notes  that  it  is  effective  for  many  computer-based 
systems.  Cost-benefit  analysis  is  sensitive  to  assumptions 
such  as  discount  rates  and  residual  value. 

If  cost-benefit  analysis  is  appropriate  or  can  be  applied, 
there  are  problems  or  weaknesses.  Chervany  and  Dickson 
(1970)  note  that,  a  major  source  of  difficulty  is  the  failure 
to  view  Management  Information  System  development  and  evalua¬ 
tion  as  a  problem  in  decision-making.  Keen  (1981)  identifies, 
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a  general  weakness  of  the  cost-benefit  approach  as  being  its 
requirement  for  knowledge,  accuracy  and  confidence  about 
issues  which  for  innovations  are  unknown,  ill-defined  and 
uncertain.  The  complex  calculations  of  cost-benefit  analysis 
are  replaced  in  value  analysis  by  rather  simple  questions 
about  usefulness. 

King  and  Schrems  (1978),  highlight  three  problems  with 
using  cost-benefit  analysis  for  evaluating  decisions  about 
information  system  in  organizations.  First,  the  difficulties 
of  assigning  cost  and  benefits,  second,  the  failure  to  specify 
the  critical  characteristics  demanded  of  the  system  and 
finally,  social  and  political  realities. 

There  are  several  methods  suggested  in  the  literature  for 
conducting  cost-benefit  analysis;  the  most  common  methods  are 
discussed  in  the  following  paragraph.  Chervany  and  Dickson 
(1970),  conclude  that: 

It  is  the  prime  responsibility  of  the  MIS  specialist  to 
synthesize  these  three  elements  -  controllable  variables, 
constraints,  and  uncontrollable  variables  -  and  project 
payoff  functions  (costs  and  benefits)  for  the  alternative 
MIS  configurations  under  consideration. 

They  further  note  that  cost-benefit  analysis  are  the 
results  of  decisions,  of  not  time;  therefore,  techniques  of 
economic  evaluation  of  management  information  systems  must  be 
decision-activity  oriented. 

B .  USER  SATISFACTION 

This  section  summarizes  comments  and  observations  from 
several  authors  on  user  satisfaction.  First,  a  discussion  on 
productivity  gains  that  can  be  addressed  by  Management 
Information  System  is  presented.  Next,  effectiveness  and 
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efficiency  for  users  information  system  are  discussed. 
Finally,  some  of  the  expected  payoffs  and  costs  are  des¬ 
cribed. 

Chandler  (1982)  states: 

IS  can  be  evaluated  from  two  different  perspectives: 

•  Focus  on  the  computer  system  domain,  where  performance 
is  measured  in  terms  of  resource  utilization,  cost  and 
efficiency . 

•  Focus  on  user  domain  throughput,  where  reliability  and 
response  time  are  common  measures. 

An  information  system  can  be  viewed  as  a  symbiotic 
relationship  between  the  users  of  the  system  and  the  system 
itself.  Ideally,  an  information  system  should  be  evaluated 
with  equal  consideration  given  to  both  user  constraints  and 
to  system  constraints . 

1 .  Productivity 

Amerrezvani  (1986)  said,  "the  recent  explosion  of 
information  technology  has  loosed  a  Pandora's  box  of  questions 
about  how  computers  and  electronic  communications  can  be  used 
effectively  in  the  workplace,  however,  current  uses  of  PCs 
don't  always  contribute  markedly  to  productivity". 

Olson  and  Lucas  (1982)  point  out  that,  "Office 
Automation  is  expected  to  increase  organizational  productivity 
through  redefinition  of  office  work  rather  than  increase 
efficiency  of  current  office  functions."  Additionally  they 
state  that,  "Automated  office  systems  can  provide  a  powerful 
mechanism  for  increasing  productivity  and  improving  the 
quality  of  work  life  by  changing  the  fundamental  nature  of 
organizational  information  processing." 

Finally,  Strassmann  (1985)  concludes  that: 
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The  cost  of  automating  office  is  a  matter  of  people  not 
a  matter  of  machines.  OA  projects  are  dominated  by  user 
costs,  or  organization  cost.  Due  to  heavy  initial  orga¬ 
nizational  costs,  the  benefits  from  most  investment  in  IT 
can  be  realized  in  the  long  run.  Realizing  a  long-term 
payoff  from  IT  depends  on  the  gain  one  obtains  from  people 
who  learn  how  to  work  more  effectively. 

The  primary  objective  of  such  projects  is  to  rapidly  make 
people  much  more  productive.  And  do  not  automate 
obsolete  patterns  of  work,  because  you  will  not  get  the 
full  benefit  of  the  technology.  You  will  only  get  minor 
gains  and  minor  gains  soon  will  no  longer  be  sufficient 
to  maintain  a  competitive  edge. 

2.  Effectiveness /Efficiency 

Ives,  et.al.  (1983),  state  that: 

DSS  can  be  judged  on  two  criteria:  efficiency  and 
effectiveness.  Efficiency  deals  with  how  well  they 
do  what  they  do.  While  effectiveness  takes  a  broader 
focus.  Are  they  doing  the  right  thing?  Are  new 
computer  technologies  being  successfully  integrated 
into  the  organization? 

Strassmann  in  (1985)  "Making  Office  Systems  Pay  Off" 
(1985) ,  prefers  to  measure  effectiveness  -  doing  the  right 
thing.  Effectiveness  relates  to  the  operation  of  groups,  not 
individual,  and  effectiveness  is  judged  by  customer 
satisfaction.  A  higher  level  of  payoffs  come  from  improving 
group  effectiveness  and  efficiency. 

3 .  Payoff a/Costa 

Strassmann  (1985),  states  that: 

The  goal  of  office  automation  from  the  point  of  view 
of  many  top  managements,  is  to  reduce  employee  head- 
count  or  to  reduce  direct  expenses.  These  expenses 
include  telephone  charges,  courier  service,  travel, 
etc-  Management  wants  bottom  line  payoffs.  Office 
automation  can  no  longer  be  justified  by  promoting 
the  use  of  technology  with  the  vague  claim  that  it 
will  "increase  productivity." 
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Levels  of  payoffs.  There  appears  to  be  a  progression 
of  potential  payoffs  from  installing  office  automation 
systems.  The  lowest  level  payoffs  come  from 
automating  individual  tasks,  aimed  at  reducing  head- 
count.  These  payoffs  probably  will  not  be  great 
because  they  involve  individual  tasks  and  individual 
employees.  A  higher  level  of  payoffs  come  from 
improving  group  effectiveness  and  efficiency. 

Getting  a  financial  payoff  from  automated  office 
systems  will  not  be  easy.  New  technology,  by  itself, 
is  not  the  answer.  The  big  payoffs  will  come  from 
solving  the  "people"  problem. 


Additionally,  Strassmann  in  (1985)  "The  Real  Cost  of 
OA",  identifies  six  cost  and  payoff  characteristics  associated 
with  Office  Automation  (OA) .  The  prime  responsibility  of 
management  on  information  technology  projects  is  to  cope  with 
organizational  issues  rather  than  with  technology  issues. 

•  Organized  labor  is  becoming  keenly  aware  of  the  negotia¬ 
ting  leverage  that  mismanaged  OA  projects  hand  over  to 
them. 


•  The  cost  of  any  activity  that  is  labor  intensive  tends 
to  rise  faster  than  the  general  rate  of  inflation. 

•  Sending  employees  to  classes  on  company  time  at  a 
company  training  center  is  the  most  expensive  way  of 
learning . 

•  Most  importantly,  the  risks  of  run-away  organizational 
costs  due  to  technical  system  failures  cannot  be 
tolerated . 

•  To  create  the  right  environment  for  sharing  organization 
learning,  it  will  be  necessary  to  create  new  standards 
for  access  to  information  sources. 

•  Until  we  have  better  insights  into  the  behavioral 
aspects  of  people,  organizational  costs  will  be  high. 
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Olson  and  Lucas  (1982),  state  that: 

The  political  view  assumes  that  automation  implies  a 
division  of  labor  that  leads  to  decrease  in  skill, 
knowledge,  and  worker  control  over  activities. 

C .  OTHER  MODELS 

This  section  discusses  other  models  or  perceptions 
collected  in  the  nonempirical  literature  review.  They  are 
broken-down  into  pieces  that  address  productivity,  effecti¬ 
veness/efficiency  and  other  general  perspective. 

1 .  Productivity 

Borko  (1983)  defines  productivity  as  the  ratio  of  the 
number  of  units  of  output  to  the  number  of  units  of  input. 
Productivity  can  be  measured  by  converting  input  and  output 
units  to  their  respective  dollar  equivalents.  He  further 
states  that: 

Productivity  rises  when  the  number  of  units  of  output 
increases  while  input  holds  steady;  or  conversely,  when 
the  number  of  units  of  output  remains  fixed  and  the  number 
of  units  of  input  decreased. 

Obstacles  to  measuring  knowledge  worker  productivity: 

•  Defining  the  nature,  the  value,  and  the  unit  of  measure 
for  the  output  of  the  knowledge  worker. 

•  Tendency  to  deal  with  only  the  quantifiable  and  not  with 
the  quality  of  the  outputs. 

•  The  work  done  by  a  knowledge  worker  in  ?  current  time 
period  may  not  show  results  until  several  periods  later. 


Carlyle  (1987),  states,  "The  difficulty  of  measuring 
productivity  gains  has  also  deterred  potential  buyers,  but 
some  vendors,  like  Texas  Instruments,  are  in  it  for  the  long 
haul."  He  further  notes  that,  at  a  time  when  the  industry 
is  paying  lip  service  to  the  need  for  Computer  Aided  Software 
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Engineering  (CASE) ,  computerized  tools  for  software  developers 
are  emerging  ahead  of  an  accepted  productivity  index  or 
effective  measure.  Many  large  companies  are  afraid  that  they 
could  invest  more  in  trying  to  improve  productivity  than  they 
would  actually  be  able  to  save. 

Mason,  et.al.  (1984)  said,  "In  the  area  of  public 
productivity  and  computer  use,  computers  serve  three  purposes; 
reducing  costs  ... increasing  revenues .. .most  impor¬ 
tant ..  .providing  better  services.” 

2.  If f*ctiven**»/Eff iciency 

Hamilton  and  Chervany  (1981),  contend  that,  effec¬ 
tiveness  is  determined  by  comparing  performance  to  objectives. 
The  way  to  assess  system  effectiveness  is  to: 

•  Determine  the  task  objective  of  the  system  or  the 
organizational  units  [using]  the  system  and  to  develop 
criterion  measures  to  assess  how  well  the  objectives 
are  being  achieved. 

System  effectiveness  is  determined  by  attainment  of  a 
normative  state,  e.g.,  a  standard  for  "good"  practices. 
Effectiveness  is  conceptualized  in  terms  of  resource  viability 
rather  than  in  terms  of  specific  task  objectives.  Evaluation 
of  system  effectiveness  is  difficult  due  to  its  multi- 
dimensionality,  its  quantitative  and  qualitative  aspects  and 
the  multiple,  and  often  conflicting,  evaluator  viewpoints. 

The  assessments  of  management  information  system 
effectiveness  are  often  controversial  and  can  be  sources  of 
disagreement  and  conflict  between  different  functional  groups 
involved  in  MIS  implementation:  users,  management  information 
system  developers,  internal  auditors,  and  top  management 
personnel.  Maggiolini,  (1986),  identifies  four  points  that 
can  improve  office  automation  technology  efficiency  by: 
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•  Automating  all  or  some  parts  of  the  office  information 
process; 

•  Eliminating  some  of  the  transformations  of  medium; 

•  Eliminating  or  reducing  the  shadow  activities/functions; 

•  Speeding  up  the  information  process  itself. 

3.  General  Perspective 

Hamilton  and  Ives  (1982),  said  that,  ''the  key  to  good 
research,  though  is  not  just  in  choosing  the  right  research 
strategy,  but  in  asking  the  right  questions  and  picking  the 
most  powerful  method (s)  for  answering  questions  given  the 
objectives,  research  setting  and  other  salient  factors." 
Ginzberg  (1978),  states  that;  The  key  point  measure  of 
effectiveness  depends  upon  the  intent  of  a  project  its  goal. 
We  can  develop  a  behavioral  measure  of  Operation 
Research/Effectiveness  if  we  explicitly  address  the  issue  of 
goals  at  the  start  of  the  project.  Finally,  Strassmann 

(1976),  suggests  nine  steps  to  better  management  of  Management 
Information  System: 

a.  Identify  costs 

b.  Keep  scores  on  unit  costs 

c.  Establish  standard  costing 

d.  Set  up  accountability  centers 

e.  Apply  competitive  pricing 

f.  Plan  for  the  long  term 

g.  Let  the  users  control 

h.  Deemphasize  the  technology 

i.  Use  job  enlargement 
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III.  METHODOLOGY 


A.  SELECTING  8TUDIES  OF  SYSTEM  EFFECTIVENESS 

1.  Choosing  m  Modiua  from  Which  to  Select  Studios 

Published  articles  were  chosen  as  the  source  from 
which  to  select  the  studies  of  system  effectiveness  to  be 
reviewed  in  this  thesis.  Published  transactions  of 
conferences  were  excluded  for  three  reasons: 

a.  The  difficulty  of  obtaining  comprehensive  bibliographical 
search  coverage  of  all  conferences  that  hold  sessions 
related  to  system  effectiveness.  For  example,  an  on-line 
bibliographic  search  of  computer  and  management 
information  system  (MIS)  failed  to  cover  the  important 
annual  International  Conference  on  Information  Systems. 

b.  No  evidence  exists  to  support  a  decision  that  papers 
presented  at  a  conference  that  publishes  its  transactions 
are  superior  to  a  conference  that  does  not.  Leaving 
aside  the  problems  of  obtaining  reliable  bibliographic 
coverage  of  conferences  that  do  publish  transactions,  to 
include  them  would  unfairly  exclude  papers  from  non¬ 
publishing  conferences. 

c.  Both  of  the  preceding  issues  are  moot  in  that  presenting 
a  paper  at  a  conference  is,  by  convention,  a  step  to 
submitting  it  for  publication  by  a  journal.  Indeed, 
papers  accepted  by  the  editors  of  the  International 
Conference  on  Information  Systems  convey  the  best  of  them 
to  journal  editors  for  consideration  as  published  pieces 
(Gallegos, 1985) .  In  short,  conference  papers  worthy  of 
publication  will  appear  in  journal  omitting  conference 
papers  prevents  redundant  entries. 
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2.  Choosing  Journals  from  Which  to  Salact  Studios 

This  review  of  academic  studies  began  with  a  selection 
of  journals  from  which  reports  of  these  studies  would  be 
taken.  All  reports  of  studies  of  system  effectiveness  in 
English-language  journals  in  the  field  of  management 
information  systems  (MIS)  and  the  related  field  of  management 
science  were  surveyed.  We  took  as  an  authority  for  choosing 
this  set  of  journals  the  works  of  Davis  (1980),  Hamilton  and 
Ives  (1983)  and  Vogel  and  Wetherbe  (1984) .  A  consensus  of 
journal  names  from  these  studies  yields  the  following  list: 

a .  Academy  of  Management  Journal 

b .  Accounting  Review 

c .  ACM  Computing  Surveys* 

d.  Administrative  Science  Quarterly 
e •  Communications  of  the  ACM 

f .  Data  Base 

g .  Datamation 

h.  Decision  Sciences 

i .  EDP  Analyzer** 

j .  Harvard  Business  Review 

k .  IEEE  Computer* *  * 

l .  Information  and  Management 

m.  Interfaces 

n .  Journal  of  Management  Information  Systems 

o .  Journal  of  Systems  Management 

p.  Management  Information  Systems  Quarterly 

q .  Management  Science 

r .  Omega 

s .  Sloan  Management  Review 

*  ACM:  Association  for  Computing  Machinery 
**  EDP:  Electronic  Data  Processing 

***  IEEE:  Institute  of  Electronic  and  Electrical  Engineering 
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Hereafter,  this  list  is  known  as  the  consensus  list.  To  it, 
was  added  Public  Productivity  Review  which  was  discovered 
during  a  search  for  periodicals  related  to  the  issue  of 
productivity.  While  the  preferred  publication  outlets  on  the 
consensus  list  are  refereed  journals  written  by  and  for 
academic  members  of  the  MIS  community,  the  list  includes 
periodicals  that  are  aimed  at  an  audience  of  MIS  practitioners 
and  some  that  are  not  journals  per  se  but  are  commercially 
published  trade  publications. 

3.  Choosing  a  Tima  Span  from  Which  to  Salact  Studios 

The  work  on  this  review  began  in  January,  1989  and  was 
completed  by  June,  1989.  The  last  journal  articles  that  were 
considered  for  inclusion  were  published  with  the  calendar 
period  of  1988.  1970  was  chosen  as  the  beginning  point  for 
on-line  and  manual  bibliographic  searches. 

However,  no  time  limit  was  placed  on  inclusion  of 
studies  that  were  discovered  by  virtue  of  references  in 
articles  found  in  the  1970  to  1988  range  by  manual  or  on-line 
searches.  The  assumption  was  that  articles  worthy  of 
inclusion  that  were  published  prior  to  1970  would  be 
referenced  in  articles  published  after  1970. 

4 .  Searching  for  Studiaa  of  System  Effectiveness 

a.  On-Linm  Bibliographic  Saarch 

In  January,  1989,  the  Dialogueue  access  station  in 
the  Dudley  Know  Library  of  the  Naval  Postgraduate  School  was 
used  for  a  bibliographic  search  based  on  key  words  in  the 
titles  and  abstracts  of  articles.  The  following  search  key 
was  constructed: 
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(Productivity  or  Bffoctivonoas)  and  (Measures  or 
Measureaent  or  Assess  or  Assessment)  and  Information 
Systama . 

This  search  identified  154  items  that  matched  the 
search  key.  Abstracts  on  the  128  of  these  that  were  published 
in  English  were  requested.  Out  of  that  set,  51  items  were 
identified  that  dealt  with  the  productivity  or  effectiveness 
of  information  systems.  The  other  77  items  dealt  with  the 
monitoring  of  computer  performance,  productivity  of  software 
development  or  topics  unrelated  to  system  effectiveness  but 
nevertheless  included  some  combination  of  the  search  key  words 
in  their  titles  or  abstracts . 

Of  the  51  items  related  to  system  effectiveness, 
26  were  conference  papers.  Out  of  the  25  items  dealing  with 
productivity  or  effectiveness  of  information  systems  that  were 
journal  papers,  ten  were  in  journals  on  the  consensus  list. 
This  search  covered  several  important  confer  ices  related  to 
information  systems,  including  some  of  which  we  would  not 
otherwise  have  known. 

Nonetheless,  as  mentioned  earlier,  the  flagship  MIS 
conference,  the  annual  International  Conference  on  Information 
Systems,  was  not  represented.  In  retrospect,  this  on-line 
search  was  naive  in  its  conception  and  incompetent  in  its 
execution.  During  the  manual  bibliographic  search,  it  was 
discovered  that  the  "productivity"  and  "effectiveness" 
keywords  were  scarcely  comprehensive  in  the  coverage  of  the 
literature  of  system  effectiveness. 

The  keywords  that  authors  in  this  literature 
applied  to  their  works  included: 

-  Cost 

-  Benefit 
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-  Cost  Benefit 

-  Evaluation 

-  Success 

-  Utilization 

-  Performance 

Moreover,  the  search  key  applied  only  to  Dialogue  file  13, 
which  is  the  INSPEC  data  base  for  the  period  from  1977  to 
1988.  That  this  file  began  its  coverage  at  1977  rather  than 
1970  already  meant  that  the  original  intention  was 
compromised.  A  different  INSPEC  file  contained  items 
published  before  1977.  That  file  would  have  been  included  in 
the  search  but  for  a  funding  limit  on  access  to  bibliographic 
search  services. 

Subsequent  experimentation  with  on-line 
bibliographic  searching  revealed  that  other  databases  in  the 
Dialogue  set  might  have  yielded  more  articles.  Certainly,  by 
not  searching  beyond  the  INSPEC  file,  which  covers  the 
computer  and  MIS  fields,  publications  in  fields  such  as 
general  management  (Harvard  Business  Review,  Sloan  Management 
Review)  and  management  science  (Management  Science,  Decision 
Science,  Administrative  Science  Quarterly)  were  missed. 

Because  limited  funding  prevented  further  on-line 
bibliographic  searching  with  a  wider  net  of  key  words  applied 
a  broader  set  of  databases,  this  study  turned  to  manual 
searching . 

Jb.  Manual  Bibliogrmphic  Search 

We  manually  examined  the  Periodicals  on  the 
consensus  list  for  the  years  1970  to  1988  were  manually 
examined  (although  some  of  these  periodicals  were  founded 
after  1970)  .  Indices  of  article  titles  for  the  covered  years 
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were  examined  or  every  issue  of  a  periodical  (where  indices 
were  not  available)  were  examined.  The  manual  search  sought 
titles  of  articles  that  seemed  to  deal  with  the  effectiveness 
of  information  systems.  This  included  articles  on  the 
performance  of  systems  in  specific  industries  or  by  type  of 
specific  application  (such  as  decision  support  systems)  as 
well  as  information  systems  in  general.  This  effort  yielded 
93  articles  purporting  to  deal  with  the  measured  effectiveness 
of  information  systems. 

The  manual  search,  not  surprisingly,  found  the  same 
ten  titles  that  were  revealed  by  the  on-line  search.  The 
number  of  articles  identified  in  the  on-line  search  compared 
to  the  far  greater  number  turned  up  by  the  manual  effort  was 
dismaying.  This  was  the  point  at  which  it  was  discovered  how 
hopelessly  limited  had  been  the  key  word  synonyms  for 
productivity  and  effectiveness  that  were  used. 

c.  Rmfmrmncma  from  Journal  Articlmm 

Reference  lists  at  the  end  of  articles  found 
manually  were  used  to  track  down  important  works  that  either 
were  published  before  1970  or  whose  relevance  was  disguised 
by  its  titles.  This  procedure  produced  an  additional  31 
articles.  These  included  works  that  dealt  with  system 
effectiveness  as  well  as  articles  that  dealt  with 
productivity,  cost-benefit  analysis,  organizational 
effectiveness  and  research  methods  that  were  written  for  a 
broader  audience  than  the  MIS  discipline. 
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d.  Gathering  of  Articl*  Reference  by  Other  Means 

When  studying  a  particular  topic,  a  researcher 
becomes  sensitized  to  its  appearance  in  any  form.  Discussions 
of  system  effectiveness  or  productivity  in  popular  or  trade 
publications,  that  might  otherwise  go  unnoticed,  will  now  leap 
into  a  researcher's  awareness.  Even  colleagues  who  are 
peripheral  to  a  study  become  sensitized  as  well.  They  offer 
references  and  clippings  dealing  with  productivity  and  the 
effectiveness  of  information  systems.  By  these  processes  the 
inventory  of  works  on  productivity,  system  effectiveness  and 
the  measurement  of  effectiveness  in  this  study  was  built. 

A  good  deal  of  these  were  from  popular  and  trade 
publications.  They  included  magazine  articles,  newspaper 
items,  student  theses,  books,  transcripts  of  speeches  and 
promotional  literature  on  software  and  consulting  services. 

5 .  Sorting  the  Works  on  System  Effectiveness 

In  combination,  the  preceding  methods  yielded  an 
overall  list  of  118  references  that  was  initially  considered, 
by  virtue  of  their  titles,  for  review  in  this  thesis.  They  are 
listed  in  Appendix  A. 

s.  Empirical  Studies 

A  reading  of  the  118  articles  in  Appendix  A 
identified  45  empirical  studies  of  system  effectiveness. 
These  45  articles  report  empirical  studies  of  the 
effectiveness  of  an  information  systems  or  an  aspect  of 
research  techniques  for  studying  system  effectiveness. 

These  articles  are  distinguished  from  others  in  the 
original  set  of  118  in  that  they  report  findings  based  on  data 
collected  specifically  for  the  study  being  reported.  These 
are  the  studies  that  this  thesis  evaluates.  They  are  studies 


are  listed  in  Appendix  B.  Among  these  45  were  articles  that 
did  not  deal  directly  with  a  study  of  a  system  implementation 
but  rather  were  concerned  with  the  development  of  a  research 
tool  to  3tudy  implementation  success.  These  were  retained 
because  their  study  research  techniques  contributes  to  how  we 
know  what  we  know  about  system  effectiveness. 

Excluded  from  this  review  were  studies  that  dealt 

with : 

1.  Computer  performance. 

2 .  Software  development . 

3.  Other  topics  not  related  to  information  system 
effectiveness . 

b.  Non -Bmpiri cal  Works  About  the  Study  of  Syatmm 

Sffoctivmnmas 

Excluded  from  this  review  of  empirical  studies  were 
articles  which  did  not  report  findings  of  a  concrete  study  of 
a  system  implementation.  These  included  essays  on  system 
effectiveness,  literature  reviews  of  system  effectiveness  or 
prescriptions  for  the  methods  of  studying  information  system 
effectiveness.  Appendix  C  lists  the  22  articles  in  this 
group.  They  are,  of  course,  the  references  that  were 
discussed  in  the  preceding  chapter  which  reviewed  the  MIS 
literature  on  the  assessment  of  system  effectiveness. 

c.  Miacollanmous  Literature 

The  remaining  references  from  the  initial  118  were 
categorized  as: 

Popular  and  trade  literature. 

Literature  dealing  with  productivity  or  organizational 
effectiveness  in  general  but  not  specifically  related  to 
MIS. 


Works  dealing  with  research  methodology  without  specific 
application  to  MIS. 

These  are  listed  in  Appendix  D. 

B.  REVIEWING  THE  EMPIRICAL  STUDIES 

The  45  empirical  studies  (listed  in  Appendix  B)  were 
individually  evaluated  for: 

1.  Design  of  data  collection 

2.  Handling  of  threats  to  internal  validity 

3.  Handling  of  threats  to  external  validity 

4.  Time  dimension 

5.  Data  collection  methods 

6.  Definition  of  effectiveness 

1 .  Design  of  Data  Collection 

From  a  reading  of  the  descriptions  of  research 
methodology  and  data  collection  techniques  in  each  study,  it 
can  be  determined  if  it: 

a.  Employed  an  explicit,  experimental  variable. 

b.  Gathered  data  before  the  administration  of  an 
experimental  variable. 

c.  Gathered  data  after  the  administration  of  an  experimental 
variable . 

d.  Gathered  data  from  comparison  groups  at  the  same  points 
in  time. 

e.  Whether  the  membership  in  comparison  groups  was 
randomized . 

Knowledge  of  these  five  elements  permitted  classification  of 
each  study  according  to  its  research  design  using  the  Campbell 
and  Stanley  (1966)  taxonomy. 
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2.  Handling  Thraata  to  Intamal  Validity 

Once  a  study  was  classified  according  to  the  Campbell 
and  Stanley  taxonomy,  it  was  possible  to  assess  its  ability 
to  handle  each  of  the  threats  to  internal  validity  listed 
below: 

a .  Eimtory 

A  history  effect  involves  events  occurring  between 
a  pretest  observation  and  a  posttest  observation  which  affect 
an  observed  phenomenon  along  with  an  intervening  variable  X. 
Designs  that  collect  data  at  both  pretest  and  posttest  points 
but  lack  a  parallel  control  group  cannot  control  for  the 
effect  of  history  on  a  study's  findings.  Designs  lacking  a 
pretest  point  but  involving  a  comparison  group,  while  weak  in 
other  respects,  can  control  for  the  effect  of  history. 

b.  Maturation 

Maturation  refers  to  changes  within  the  respondents 
themselves  that  can  account  for  differences  in  measurements 
between  the  first  and  second  observations.  As  with  the  history 
effect,  designs  that  collect  data  at  both  pretest  and  posttest 
points  but  lack  a  control  group  cannot  control  for  the  effect 
of  maturation.  Where  a  design  lacks  a  pretest  point  but 
involves  a  comparison  group,  the  capacity  of  the  design  to 
exclude  maturation  as  a  rival  explanation  of  its  findings  in 
not  clear  from  a  design  alone  (as  in  the  case  of  the  static 
group  design) .  Designs  that  use  randomized  comparison  groups 
(such  as  the  posttest  only  control  group)  are  not  affected. 
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c.  Re-test  Sophistication 

Re-test  sophistication  arises  when  respondents 
become  skilled  at  taking  a  test,  filling  out  a  questionnaire, 
answering  in  an  interview  or  taking  part  in  an  experiment.  As 
much  as  the  intervening  variable,  such  changes  in  skill  can 
cause  changes  in  readings  at  a  second  observation.  Where  a 
design  lacks  a  pretest  phase,  yet  uses  comparison  groups,  this 
effect  is  controlled.  Designs  that  involve  a  pretest  and 
posttest  but  lack  a  comparison  group  (such  as  the  one  group 
pretest/posttest  design)  are  threatened  by  re-test 
sophistication.  For  other  designs,  the  effect  is  neither 
controlled  nor  threatening. 

d.  Instrument  Calibration 

Where  people  are  the  instruments  of  recording  and 
measuring,  changes  within  them  can  produce  changes  in  tne  in 
reports  of  events.  Examples  of  human  re-calibration  are 
fatigue,  growing  skill  at  conducting  interviews,  boredom  or 
developing  self  confidence  as  observers.  This  effect  is 
controlled  in  designs  that  involve  comparison  groups  with 
randomized  assignment  of  members  or,  lacking  that,  have  no 
pretest  stage.  The  one-group  pretest/posttest  design  is 
vulnerable  to  instrument  calibration  effects. 

•.  Statistical  Regression 

Where  comparison  groups  in  a  study  of  system 
implementation  have  been  selected  precisely  for  their  extreme 
differences  in  scores  on  tests  or  surveys  before  a  study,  the 
statistical  regression  toward  the  means  score  is  likely  to 
occur  on  the  posttest  observation.  Generally,  respondents  with 
extreme  scores  on  one  occasion  will  not  produce  the  same 
extremes  on  another  occasion.  A  design  alone  does  not  tell 
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whether  statistical  regression  affects  a  particular  instance 
of  a  one-group  pretest /posttest  design.  The  static  group 
design,  however,  is  vulnerable  to  is  effect.  Designs  involving 
randomized  comparison  groups  are  able  to  control  for  the 
effect  of  regression. 

t .  Selection  Bias 

Selection  bias  is  a  rival  explanation  for  a  study's 
findings  when  comparison  groups  assumed  to  be  equivalent  are 
actually  different  due  to  biases  operating  during  the 
selection  of  respondents.  These  differences  can  account  for 
the  differences  in  reported  data  between  the  two  groups.  Any 
design  with  a  pretest  and  a  posttest  can  control  for  this 
effect.  Those  lacking  a  pretest  or  a  randomized  comparison 
group  are  vulnerable  to  selection  bias. 

g.  Respondent  Mortality 

Between  non-randomized  comparison  groups,  there 
can  be  a  selective  drop-out  of  participants  from  a  study  while 
researchers  assume  that  dropping-out  is  random.  This 
differential  drop-out  effect  can  account  for  differences  in 
data  reported  from  the  groups.  The  designs  that  control  for 
this  effect  and  those  that  a  -*5  vulnerable  to  it  are  identical 
to  that  of  the  effect  of  selection  bias. 

h.  Selection-Maturation  and  Other  Interactions 

In  multi-group  research  designs,  the  interaction 
of  a  selection  bias  with  a  maturation  effect  could  be  mistaken 
for  the  effect  of  an  intervening  experimental  variable. 
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3.  Handling  Threats  to  External  Validity 

As  with  the  threats  to  internal  validity,  knowledge 
of  a  study's  placement  in  the  Campbell  and  Stanley  (1966) 
taxonomy  of  research  designs  allowed  researchers  to  assess 
its  capacity  to  exclude  threats  to  external  validity. 

a.  Zntoroction  of  Tooting  ond  on  X  Variable 

In  certain  kinds  of  studies,  the  experience  of 
participating  in  an  experiment,  completing  a  questionnaire, 
submitting  to  an  interview  or  being  observed  at  pretest  stage 
sensitizes  a  group  of  respondents  to  an  X  variable.  That  is, 
by  being  subjects  in  a  study,  these  people  have  a  heightened 
sensitivity  to  an  X  stimulus  that  is  not  characteristic  of  the 
general  population. 

To  that  extent,  the  findings  of  such  a  study  are 
not  generalizable  beyond  the  sample  group.  The  one  group 
pretest/posttest  design  and  the  true  experimental  design  are 
both  vulnerable  to  this  effect.  Groups  that  have  some  elements 
of  non-pretest  comparison  such  as  the  posttest  only  control 
group  and  the  Solomon  four-group  design  are  able  to  control 
for  it.  Other  designs  are  not  affected  either  way. 

b.  Intormction  of  Soloction  ond  t ho  X  Vorioblo 

Campbell  and  Stanley  (1966)  use  as  an  example  of 
this  threat  a  researcher  seeking  a  school  district  to  provide 
a  sample  of  students  for  a  study.  Nine  districts  turn  down  the 
researcher.  A  tenth  district  agrees  to  participate.  Surely, 
in  some  way,  the  tenth  differs  from  the  other  nine.  The 
results  that  this  study  yields  will,  therefore,  not  be 
generalizable  to  a  large  population  of  students  per  se. 

To  the  extent  that  an  experimental  variable  in  this 
study  has  differential  effects  between  an  experimental  group 
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and  a  control  group,  it  is  likely  that  characteristics  of  the 
school  district  caused  the  X  variable  to  be  more  successful 
than  it  would  begin  other  school  districts. 


c.  Aemctive  Arrangement* 

The  artificiality  of  an  experimental  setting  or  of 
questionnaire  choices  put  respondents  in  a  Guinea-pig,  I-am- 
special  role.  Whatever  behavior  that  is  thereby  observed  or 
recorded  does  not  represent  how  these  same  people  are  likely 
to  act  in  everyday  life. 

d.  Multiple  X  Interference 

While  a  conventional  research  design  involves  a 
single  administration  of  an  X  variable  to  experimental  group, 
some  quasi-experimental  designs  such  as  the  equivalent  time 
samples  design,  and  the  equivalent  materials  samples  design, 
involve  repeated  administrations  of  an  X  variable  to  one  group 
of  respondents. 

The  trouble  is  that  the  effects  of  prior 
administrations  remain  with  the  respondents,  building  up  with 
each  subsequent  administration.  Such  groups  undergo  an 
experience  as  research  subjects  that  has  few  counterparts  in 
everyday  life.  Thereby,  they  are  rendered  unrepresentative  of 
the  larger  population. 

4 .  Tim*  Dimension 

The  Campbell  and  Stanley  (1966)  taxonomy  of  research 
designs  also  provides  a  means  to  assess  whether  a  study  has 
taken  account  of  the  time  dimension  as  a  basis  for 
establishing  causal  inference. 

Studies  employing  pretest  and  posttest  observations 
and  those  that  employ  time  series  multiple  observations,  by 
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definition,  account  for  the  time  dimension.  Studies  involving 
an  observation  at  a  single  point  in  time  do  not. 

5.  Technique*  of  Data  Collection 

By  convention,  an  empirical  study  is  obliged  to 
describe  the  technique  by  which  it  gathers  data.  That 
convention  was  upheld  by  all  45  of  the  studies  reviewed  here. 
Each  study  was  classified  according  to  the  following  taxonomy 
of  data  techniques: 

a.  Self-completion  questionnaire,  administered  under  the 
supervision  of  the  researchers. 

b.  Self  completion  questionnaire,  mailed  out  to  subjects  to 
be  returned  by  mail. 

c.  Interview  questionnaire. 

d.  Telephone  interview. 

e.  Laboratory  experiment. 

f.  Non-obtrusive  technique:  archival  records. 

g.  Review  of  observations  by  expert  panel. 

6.  Definition  of  System  Effectiveness 

Each  study  was  read  for  its  definition  of  how  a  system 
would  be  judged  to  be  effective.  No  taxonomy  of  such 
definitions  exists  a  priori  The  classification  categories 
emerged  in  the  process  of  reviewing  the  studies. 

7 .  Degree  of  Change  in  Effectiveness  Reported  in  the 

System 

This  thesis  suggests  that  the  appropriate  enterprise 
for  academic  research  in  MIS  effectiveness  is  the  construction 
of  a  formal  theory  of  system  effectiveness.  A  theory  of  system 
effectiveness  necessarily  involves  a  knowledge  of  the 
determinants  of  such  effectiveness.  To  understand  the 
determinants  of  effectiveness,  a  researcher  must  know  by  how 
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much  the  effectiveness  of  an  organization  was  improved  or 
degraded  by  the  implementation  of  an  information  system. 

Therefore,  it  is  appropriate  to  inquire  as  to  the 
level  of  changes  in  effectiveness  that  are  reported  in  each 
empirical  study  of  system  effectiveness.  If  doubts  about  the 
success  of  information  systems  are  harbored  by  senior  line 
managers,  then  a  knowledge  of  the  reality  of  information 
payoffs  is  a  contributes  to  the  current  discussion.  What  do 
these  45  academic  studies  say  about  the  successful  application 
of  information  systems? 

Each  study  was  reviewed  for  indications  of  the  level 
of  reported  changes  in  system  effectiveness. 
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IV.  FINDINGS 


Appendix  E  contains  an  annotated  description  of  each  of 
the  45  studies  evaluated  here.  Appendix  F  contains  the 
detailed  tables  of  findings  for  each  aspect  of  the  studies. 
The  studies  are  listed  in  chronological  order  by  date  of 
publication. 

A.  DESIGN  OF  DATA  COLLECTION 

In  Table  F-l  (Appendix  F) ,  the  number  at  designations  at 
the  top  of  the  columns  refer  to  evaluation  elements  in 
assigning  each  study  to  a  design  category  according  to  the 
Campbell  and  Stanley  (1966)  taxonomy  as  follows: 

1.  The  study  collected  data  in  a  pretest  stage. 

2.  The  study  employed  a  comparison  group. 

3.  Respondents  in  the  study  were  assigned  to  comparison 
groups  on  a  randomized  basis. 

1 .  Non-Expcrimantml  Designs 
s.  Onm-Sho t  Csss  Study 

Table  1  distills  the  distribution  of  research 
designs  in  these  45  studies  out  of  the  detail  of  Table  F-l 
(Appendix  F) .  Table  1  shows  that  the  one-shot  case  study  is 
the  research  design  of  choice  in  MIS  studies  of  system 
effectiveness,  by  a  two  to  one  margin.  A  one-shot  case  study 
has  no  explicit  administration  of  an  intervening,  experimental 
variable  (X) .  Rather,  as  the  brackets  around  the  X  in  the 
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design  notation  indicate,  the  subjects  from  whom  data  was 
collected,  at  the  sole  observation  point  O, ,  were  assumed  to 


TABLE  1 


DISTRIBUTION  OF  RESEARCH  DESIGNS* 

This  Design 
as  a 

Campbell  Percentage 

&  Stanley  of  Studies 

Research  Design: _ Design  Notation _ Reviewed  : 


One  Shot  Case  Study 

[X]  0, 

Static  Group  Comparison 

X  O 

o 

One  Group  Pretest/Posttest 

Oi  X  0, 

Experiment 

R  O  X  O 

R  0  0 

Posttest  Only  Control  Group 

R  X  O 

R  O 

Time  Series 

0  0X0 


o 


66.7% 

17 . 8% 

4 .4% 

4 .4% 

4.4% 

2 .2% 


’These  percentages  sum  to  99.9%  due  to  rounding  errors. 


have  been  exposed  to  the  effects  of  a  non-existent 
experimental  variable.  In  place  of  explicit  exposure  to  an 
X  experimental  variable,  the  subjects  at  are  asked  to 
recall  something.  Human  recall  is  notoriously  unreliable 
(Haberman,  1963).  For  that  reason,  the  one-shot  case  study 


is  "the  weakest  form  ofsurvey  design"  (Denzin,  1970). 
Fundamental  to  the  gathering  of  scientific  evidence  is  the 
need  for  recording  differences  (Campbell  and  Stanley,  1966) . 
Scientific  testing  requires  the  making  of  comparisons.  No 
comparisons  are  made  in  one-shot  studies.  While  one-shot 
studies  often  involve  survey  instruments  that  describe  their 
face  validity  and  Cronbach  reliability  indices  in  detail,  all 
of  that  is  a  case  of  misplaced  precision. 

It  is  another  case  of  shuffling  the  psychometric 
deck  chairs  on  the  methodological  Andrea  Doria.  In  the 
estimate  of  Campbell  and  Stanley,  rival  explanations  for  the 
findings  reported  by  one-shot  studies  are  so  numerous  as  to 
render  them  nearly  useless  in  establishing  causal  inference. 

b.  Static  Group  Comparison 

As  is  shown  in  Table  1,  the  next  most  frequent 
design  in  MIS  research  on  systems  effectiveness  is  the  static 
group  comparison.  This  design  offers  the  virtue  of  some  basis 
for  examining  differences.  The  difficulty,  however,  is  in 
attributing  the  differences  to  exposure  to  an  experimental 
variable  by  one  of  the  groups.  That  is  because  membership 
in  the  comparison  groups  in  this  design  is  not  a  randomized 
assignment . 

The  findings  are  vulnerable  to  the  rival 
explanation  that  differences  between  the  groups  predated  the 
intervention  of  an  X  variable.  Where  the  comparison  groups 
are  deliberately  selected  for  their  extreme  differences,  as 
they  were  in  some  of  the  reviewed  studies,  the  rival 
explanation  gains  potency.  Another  problem  with  the  static 
group  comparison  is  the  possibility  of  biased  drop-out  by 
respondents  in  the  period  after  the  X  variable  intervened. 
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c.  Onm  Group  Prmtmmt/Pomttmmt 

The  one-group  design  has  the  virtue  of  a  deliberate 
attempt  to  measure  the  differences  between  a  group  observed 
at  Oi  and  at  02 .  Unfortunately,  its  lack  of  a  comparison  group 
leaves  it  subject  to  a  distracting  set  of  rival  explanations 
for  its  findings.  Chief  among  them  are  other  effects 
transpiring  between  0X  and  02  besides  the  X  variable. 

While  it  is  superior  to  the  one-shot  case  study, 
the  only  recognition  the  one-group  design  merited  from 
Campbell  and  Stanley  was  that  it  was  worth  doing  only  where 
nothing  better  could  be  done.  They  employ  the  one-group  as 
a  bad  example  of  research  design  that  deludes  researchers  and 
their  constituents  alike  by  its  closeness  to  a  true 
experimental  design. 

2 .  Experimental  Designs 
s .  True  Kxpmrimmnt 

The  true  experiment,  having  pretest  and  posttest 
data  collection  points  paralleled  by  randomized  comparison 
groups,  controls  for  all  threats  to  internal  validity.  The 
difficult  with  the  MIS  empirical  studies  of  ef f ectiveness  is 
that  only  four  percent  of  them  used  this  design. 

b.  Port  test  Only  Control  Group 

While  it  lacks  a  pretest  data  collection  stage, 
the  posttest  only  design  nonetheless  qualifies  as  an 
experimental  design  by  virtue  of  its  randomized  comparison 
groups.  As  with  the  true  experiment,  its  difficulty  is  the 
rarity  with  which  it  was  practiced  in  MIS  research. 
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3.  Quasi -Experimental  Design 

A  quasi-experimental  design,  a  description  coined  by 
Campbell  and  Stanley,  is  an  attempt  to  approximate 
experimental  design  in  a  field  setting  by  scheduling  the  whom 
and  when  of  measurement  even  though  the  experimental  variable 
itself  is  not  amenable  to  scheduling. 

a .  Timm  Smrimm 

While  it  might  appear  that  a  time  series  design 
shared  the  vulnerabilities  of  the  one-group  pretest/posttest, 
this  is  not  the  case.  The  repeated  measurements  before  and 
after  an  experimental  variable  controls  for  threats  to 
internal  validity  such  as  maturation,  re-test  sophistication, 
instrument  calibration,  statistical  regression  and  the 
interaction  of  selection  bias  and  maturation.  This  renders 
it  a  much  sounder  design  that  the  one-group.  Regrettably, 
only  2%  of  the  reviewed  studies  employed  a  time-series  design. 

B.  HANDLING  THREATS  TO  INTERNAL  VALIDITY 

The  detailed  evaluations  of  the  handling  of  threats  to 
internal  validity  by  each  of  the  45  studies  reviewed  are  shown 
in  Table  F-2  in  Appendix  F.  A  +1  in  a  column  means  that  a 
study  controlled  the  threat  to  internal  validity  represented 
by  the  column  heading.  A  -1  indicates  that  a  study  was 
vulnerable  to  that  particular  threat.  A  question  mark  means 
that  the  effect  of  the  threat  upon  this  study  cannot  be 
determined  solely  by  knowledge  of  the  research  design.  No 
symbol  indicates  that  a  threat  has  no  effect  on  the  study  in 
whose  row  it  occurs . 

Recall  that  in+  ernal  validity  asks  whether  reported 
findings  are  due  to  the  effect  of  an  experimental  variable  or 
result  from  causal  factors  not  accounted  for  by  the  research. 
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Where  a  study  can  control  threats  to  internal  validity,  its 
findings  resist  challenges  to  their  scientific  standing. 


TABLE  2 

THREATS  TO  INTERNAL  VALIDITY 


Effect 
to  Vulner¬ 
able 

Effect 

Cont¬ 

rolled 

Effect 

Not 

clear 

No 

effect 

History 

73% 

27% 

Maturation 

73% 

9% 

18% 

Testing 

4% 

29% 

69% 

Instrumentation 

4% 

27% 

2% 

67% 

Regression 

29% 

4% 

67% 

Selection 

84% 

16% 

Mortality 

84% 

16% 

Interaction  of 
Selection  and 
Maturation 

22% 

11% 

67% 

To  the  extent 

that  a  study 

cannot 

exclude 

the  rival 

explanations  for 

its  findings 

posed  by 

threats 

to  internal 

validity,  the  scientific  meaning  of  the  reported  data  will  be 
lost.  The  evaluations  in  Table  F-2  are  summarized  in  Table 
2.  It  reveals  that,  on  average,  slightly  over  73%  of  the  MIS 
studies  of  system  effectiveness  are  vulnerable  to  the 
likelihood  that  their  findings  can  alternat ivelv  be  accounted 
for  by  the  effects  of  history,  maturation,  selection  bias  and 
mortality . 

This  is  a  consequence  of  the  fact  that  two-thirds  of  the 
MIS  studies  reviewed  here  employ  a  one-shot  case  study  design. 


42 


No  studies  were  clearly  vulnerable  to  statistical  regression 
as  a  rival  explanation.  Moreover,  rather  few  (4%)  had  their 
internal  validity  threatened  by  re-test  sophistication  or 
instrument  calibration. 

The  picture  that  emerges  here  is  that  of  an  academic 
discipline  that  cannot  demonstrate  with  certainty  that  its 
empirical  findings  mean  what  they  say.  These  findings  erode 
a  basis  for  establishing  causal  inference  in  the  construction 
of  a  formal  theory  of  the  effective  implementation  of 
information  systems. 

C.  HANDLING  THREATS  TO  EXTERNAL  VALIDITY 

Table  F-3  in  Appendix  F  contains  a  detailed  evaluation  of 
the  handling  of  threats  to  external  validity  by  each  of  the 
45  studies  reviewed  in  this  thesis.  The  column  headings  at 
the  top  of  Table  F-3  (to  the  right  of  the  year  column) 
includes  the  two  external  validity  threats  arising  from 
interactions  between  the  experimental  variable  and 
characteristics  of  a  study's  design. 

The  first  of  these  is  the  interaction  of  re-testing  and 
an  X  variable.  The  second  is  the  interaction  of  selection 
bias  and  an  X  variable.  The  meaning  of  both  of  these  are 
explained  in  Chapter  III.  The  third  column  is  the  threat  from 
reactive  arrangements.  The  fourth  is  the  threat  from  effect 
of  multiple  exposures  to  an  X  variable.  The  meaning  of  the 
+1,  -1,  ?  symbols  the  same  as  in  Table  F-2. 

Where  internal  validity  asks  whether  the  findings  of  a 
study  can  be  trusted  to  mean  what  they  claim,  external 
validity  deals  with  the  extent  to  which  the  findings  of  a 
study  are  generalizable  beyond  the  study  to  a  larger 
population . 
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TABLE  3 

HANDLING  OF  THREATS  TO  EXTERNAL  VALIDITY 


Threat 

Effect  to 
Vulner¬ 
able 

Effect 

Cont¬ 

rolled 

Effect 

Not 

clear 

No 

effect 

Interaction  of  Re-Testing 
with  X  Variable 

11% 

4% 

85% 

Interaction  of  Selection 
Bias  with  X  Variable 

89% 

11% 

Reactive  Arrangements 

16% 

84% 

Interference  of  Multiple 
Administrations  of 

100% 

X  Variable 


Table  3  summarizes  the  detailed  evaluations  of  threats 
to  external  validity  for  each  of  the  45  empirical  studies. 
The  use  of  research  designs  in  MIS  empirical  assessments 
determines  these  findings.  With  11%  of  the  reviewed  studies 
employing  a  pretest  stage,  exactly  11%  are  found  to  have  their 
generalizability  threatened  by  a  re-test  effect.  None  of  the 
reviewed  studies  involved  repeated  administrations  of  an  X 
variable;  therefore,  none  were  vulnerable  to  multiple  X 
interference . 

However,  the  89%  of  the  studies  that  used  non-experimental 
designs  (chiefly  one-shot  case  studies)  precisely  account  for 
the  vulnerability  to  an  interaction  of  selection  bias  and  an 
X  variable.  External  validity  deals  less  with  a  fundamental 
question  of  scientific  credibility  than  it  does  with  a  simple 
question  of  relevance.  Every  study  reviewed  here  is 
vulnerable  to  a  challenge  to  the  generalizability  of  its 
findings  to  a  larger  population.  Only  the  two  posttest-only 
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studies  provided  a  positive  control  over  any  of  the  external 
validity  threats. 

If  the  findings  of  MIS  research  are  not  representative  of 
any  population  beyond  the  studies  themselves,  the  discipline 
is  flirting  with  the  likelihood  that  its  core  work  is 
irrelevant  beyond  the  pursuit  of  publication  and  tenure. 

D.  TINS  DIMENSION 

Table  F-4  in  Appendix  F  categorizes  each  of  the  45  studies 
simply  by  whether  its  research  design  permitted  an  accounting 
of  the  time  dimension  in  the  collection  of  its  data.  11%  of 
the  studies  accounted  for  time,  through  designs  such  as  the 
true  experiment,  one-group  pretest /posttest  and  time  series. 
The  other  89%  of  the  studies  employ  designs  that  collect  data 
only  at  a  single  point.  Most  of  those  are  of  the  one-shot 
case  design.  The  capacity  of  a  research  effort  to  establish 
causal  inference  is  impaired  without  a  way  to  show  that 
variable  X  occurs  before  variable  Y. 

E.  DATA  COLLECTION  METHODS 

Table  F-5  (Appendix  F)  lists  the  type  of  data  collection 
technique  employed  by  each  study.  Two  columns  of  methods  are 
shown  in  Table  F-5.  One  third  of  the  studies  in  this  review 
used  two  methods . 

In  some  cases,  two  methods  bolstered  each  other  in  an 
effort  at  triangulation.  This  happened  where  a  survey 
questionnaire  was  used  in  conjunction  with  system  performance 
data  collected  by  the  information  system  itself.  In  other 
instances,  a  first  method  was  used  as  pilot  study,  usually  for 
instrument  development. 
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TABLE  4 

DATA  COLLECTION  TECHNIQUES  FOR  PRIMARY  METHOD 

Asking  Questions  73.3% 

Questionnaire  60.0% 

(Including  mail  out- 
mail  back  surveys:  22%) 


Interview  11.1% 

Phone  survey  2.2% 

Not  Asking  Questions  26.6% 

Non-obtrusive  methods  17.8% 

(Archival) 

Laboratory  experiment  4.4% 

Diary  2.2% 

Panel  2.2% 


TABLE  5 

DATA  COLLECTION  TECHNIQUES  FOR  SECONDARY  METHOD 

Asking  Questions  80.1% 

Questionnaire  66.7% 

(Including  mail  out- 
mail  back  surveys:  27%) 

Interview  6.7% 

Semantic  Differential  6.7% 


Not  Asking  Questions 

Non-obtrusive  methods 
(Archival ) 

Simulation 


20.0% 


13 . 3% 

6.7% 


i 
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TABLE  6 

DISTRIBUTION  OF  METHODS  BY  RESEARCH  DESIGN 


TvDe  of 

Research 

Desicrn 

True 

Post 

One- 

Shot 

Static 

Grouo 

One 

Group 

Experi¬ 

ment 

Test 

Onlv 

Time 

Series 

Method 

Questionnaire 
incl  Mailed  Out) 

61.3% 

71.4% 

50.0% 

100% 

Mailed  Out 

29.0% 

50.0% 

Interview 

12.9% 

14.3% 

Phone  survey 

3.2% 

Archival 

16.1% 

14.3% 

50.0% 

100% 

Laboratory 

100% 

Diary 

3.2% 

Panel 

3.2% 

100%* 

100% 

100% 

100% 

100% 

100% 

Sums  to  99.9%  due  to  rounding  error. 

As  Table  4  shows,  over  73%  of  the  studies  reviewed  here  use 
primary  methods  that  involve  asking  people  questions  whether 
by  self-administered  questionnaire  (completed  in  group 
sessions  or  in  a  mailed-out /mailed  back  mode) ,  personal 
interview  or  by  phone  interview. 

The  problem  posed  by  this  proportion  is  that  the  focus  of 
inquiry  is  the  effectiveness  of  systems ,  not  of  individuals . 
Even  if  the  objection  is  made  that  systems,  after  all,  are 
essentially  comprised  of  people,  then  the  fallacy  of 
reductionism  is  in  play.  It  is  a  fallacy  to  study  a  level  of 
organization  by  reducing  it  to  its  constituents  elements.  If 
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reductionism  were  a  valid  approach  to  studying  the  performance 
of  organizations  or  systems,  why  stop  at  individuals? 
Individuals,  after  all,  are  comprised  of  living  organs.  By 
this  logic,  we  would  study  information  systems  by  studying 
intestines . 

One  can  follow  this  absurd  path  to  molecular  biology  and 
beyond.  While  individuals  together  comprise  systems  (or 
groups  or  teams  or  organizations),  there  is  a  qualitative 
distinction  between  the  analysis  involved  in  studying 
individuals  and  in  studying  systems  comprised  of  individuals. 
Systems  (as  with  teams,  groups,  families  or  organizations)  are 
not  merely  the  sum  of  the  discrete  individuals. 

Directing  questions  at  individuals  gathers  data  about 
individuals  but  collects  little  about  the  characteristics  of 
the  system.  This  leaves  aside  the  even  more  troublesome  issue 
about  asking  people  questions  even  when  it  is  individuals  that 
are  being  studied.  This  preference  for  asking  questions  was 
even  a  bit  stronger  among  the  secondary  methods. 

The  core  concern  with  the  people-asking  methods  is  that 
the  methods  tend  to  create  the  very  data  they  are  recording. 
Far  from  providing  a  neutral,  innocuous  means  of  gathering 
observations  about  events,  these  methods  provoke  responses 
that,  in  themselves,  involve  role-playing,  Guinea-pig  effects, 
looking  good,  passing  a  test,  socially  desirable  responses, 
fabrication,  data  inflation  and  otherwise  carrying  on  in  ways 
that  are  not  found  in  everyday  interactions  other  than  with 
organizational  or  social  superiors.  These  same  biasing  and 
distorting  factors  are  found  in  the  reactive  setting  of 
laboratory  experiments.  But  there  they  are  well  known. 

The  trouble  with  the  survey  questionnaire  is  that  it  is 
assumed  to  have  no  biasing  or  distorting  effects  when  it,  in 
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reality  is  strongly  biasing  and  distorting  (Katz,  1942;  Hyman, 
1949;  Wyatt  and  Campbell,  1950;  Parry  and  Crossley,  1959; 
Robins,  1963;  Williams,  1964;  Weiss,  1968;  Webb  et . al .  1966; 
Sechrest,  1968;  Summers  and  Hammonds,  1969;  Hambright,  1969; 
Denzin,  1970;  Phillips,  1971). 

A  final  problem  posed  by  the  method  of  asking  questions 
is  that  the  data  collected  can  only  record  the  expression  of 
attitudes.  The  asking-people  methods  are  unable  to  record 
expressions  of  behavior.  Behavior  can  only  be  observed  or 
recorded  by  observers  as  archival  data.  While  MIS  researchers 
occasionally  speak  of  dealing  with  "behavioral"  issues  or 
methods,  in  fact  they  are  not  dealing  with  behavior  at  all  but 
attitudes . 

To  use  a  questionnaire  to  ask  a  respondent  about  their 
behavior  is  not  to  gather  data  on  behavior.  Rather  the 
process  gathers  data  on  a  respondent's  attitude  toward  the 
behavior  above  which  he  or  she  is  being  asked.  This  is  a 
distinction  that  seems  largely  lost  on  the  MIS  researchers  in 
the  study  of  system  success. 

Examining  the  association  of  method  with  research  design 
holds  no  surprises.  The  non-experimental  designs  relied 
heavily  on  the  methods  that  involve  asking  questions.  But 
then  so  did  all  of  the  true  experiments.  However,  the  other 
experimental  design,  the  posttest  only  control  group,  employed 
only  a  non-asking  method.  So  did  the  single  quasi- 
experimental  design. 

A  strong  relationship  might-  be  expected  between  a  weak 
design  such  as  a  one-shot  case  study  and  a  troublesome  method 
such  as  asking  questions  of  people.  Both  evince  a  reluctance 
to  commit  to  research  efforts  that  involve  time  series  or 
attention  to  the  details  of  non-obt rusive  methods. 
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The  discussions  in  many  of  the  reviewed  studies  echoed  the 
conventional  wisdom  that  gathering  data  on  the  costs  of  an 
information  system  is  difficult  enough.  Attempting  ti  gather 
data  on  a  system's  benefits  is  regarded  as  futile.  Thusly, 
these  researchers  justify  turning  to  methods  and  designs  noted 
more  for  their  convenience  than  their  scientific  rigor. 

If  one  has  learned  only  survey  methods  as  a  graduate 
student,  one  is  unlikely  to  explore  alternative  methods  as 
junior  faculty  member  under  pressure  to  achieve  tenure.  This 
is  especially  the  case  where  those  alternative  methods  and 
designs  take  longer  and  are  more  difficult.  Career  expediency 
leaves  causal  inference  in  the  dust.  That  the  non-obtrusive 
archival  data  method  is  practiced  so  little  is  surprising  in 
that  it  is  in  the  nature  of  information  systems  to  yield 
information  even  about  their  own  performance. 

Yet,  archival  data  was  employed  as  a  primary  method  in 
fewer  than  18%  of  the  reviewed  works  (and  as  a  secondary 
method  in  only  13%)  .  As  a  non-obtrusive  method,  archival  data 
offer  the  supreme  advantage  of  having  the  least  biasing  or 
distorting  effects. 

7.  DEFINITIONS  OF  EFFECTIVENESS 

Table  F-6  in  Appendix  F  contains  a  list  of  the  definition 
of  system  effectiveness  gleaned  from  each  reviewed  study. 
These  are  combined  and  summarized  in  Table  7  below. 

The  first  grouping,  corporate  scores,  represents  the 
traditional  measures  of  success  in  the  private  sector.  Among 
the  reviewed  studies,  they  are  a  minor  fragment  of  the 
definitions  employed  for  information  success.  The  next 
traditional  measure  of  system  effectiveness,  efficiency  (cost 
reduction  in  this  case)  was  found  in  13%  of  the  studies. 
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It  is  a  definition  how  much  under  attack  from  the  user 
attitude  enthusiasts. 

TABLE  7 

DEFINITIONS  OF  EFFECTIVENESS  FOR  INFORMATION  SYSTEMS 
Corporate  score  4% 


Profit 

2% 

Return  on  investment 

2% 

Efficiency 

13% 

Cost  reduction 

13% 

Intermediate  goods 

29% 

Information  usage 

5% 

Plan  implementation 

2% 

System  usage 

18% 

Job  factors 

2% 

Office  automation 

2% 

Effectiveness 

9% 

Decision  quality 

7% 

Performance  factors 

2% 

Value  Assessment 

7% 

Value  of  information 

5% 

System  value 

2% 

By-product 

34% 

User  attitude 

34% 

Miscellany 

2% 

Improved  grammar 

2% 

29%  of  the  studies  counted  as  a  system  output  a  factor 
that  is  actually  a  system  input  (described  in  Table  IV- 6  as 
an  intermediate  good) .  The  mistake  in  these  cases  was  to  see 
the  usage  of  information  or  a  system  as  evidence  of  the 
system's  success.  The  correct  question  is  whether  results 
from  the  usage,  not  the  level  of  usage  itself,  benefit  an 
organization.  While  there  is  much  discussion  in  the  MIS 
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literature  on  measurement  about  the  need  to  focus  on 
organizational  effectiveness  rather  than  conventional  cost¬ 
cutting  efficiency,  only  9%  of  the  reviewed  studies  employed 
effectiveness  as  a  definition  of  success. 

The  major  definition  of  system  effectiveness  found  in 
these  studies  was  some  expression  of  user  attitudes.  This 
approach  holds  that  if  people  say  they  are  happy  with  a  system 
then  the  system  is  a  success.  That  seems  sensible  enough. 

Yet,  the  popular  and  trade  literature  reviewed  in  Chapter 
I  found  that  the  payoff  from  information  systems  is  either 
falling  or  not  advancing  commensurate  with  the  investment  made 
in  it .  A  user's  attitude  about  a  system  is  an  entity  quite 
separate  from  the  economic  contribution  of  a  system  makes  to 
an  organization.  The  assumptions  in  this  approach  take  either 
of  these  two  forms: 

1.  A  user  is  satisfied  with  a  system  that  is  productive. 
A  user's  satisfaction  is  a  surrogate  measure  for  a 
system's  productivity. 

2.  Technical  efficiency  and  economic  contribution  are  not 
the  point.  At  best,  they  are  the  means.  User 
satisfaction  is  the  end. 

In  the  first  case,  the  possibility  exists  that  a  user  can  be 
satisfied  with  an  unproductive  system.  The  satisfaction  of 
a  user  may  not  necessarily  be  based  on  a  system's  efficiency 
but  on  other  factors  such  as  job-enlargement,  better  equipped 
work  areas,  learning  of  new  skills  or  protection  from 
redundancy . 

In  any  case,  system  efficiency  and  performance  must  be 
examined  apart  from  the  attitude  of  users.  It  would,  for 
example,  be  interesting  to  examine  situations  in  which  users 
were  highly  satisfied  with  unproductive  information  systems. 
That  interest  can  be  pursued  only  if  satisfaction  and  system 
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performance  remain  distinct.  The  second  case  represents  a 
matter  of  ideology,  most  often  among  the  organizational 
development  group  that  has  emerged  from  organizational 
psychology.  The  root  of  the  user-attitude  definition  of 
effectiveness  may  be  methodological. 

If  a  group  is  committed  to  the  method  of  asking  people, 
then  the  resulting  answers  are  arrogated  to  an  end.  If  you 
cannot  measure  effectiveness  at  the  system  level  then  it  is 
expedient  to  dismiss  it  as  a  focus  of  inquiry.  Can  a  formal 
theory  of  information  system  effectiveness  be  constructed  on 
a  foundation  of  attitudes  about  a  system  rather  than 
measurements  directly  from  a  system  itself? 

G.  REPORTED  CHANGE  IN  SYSTEM  EFFECTIVENESS 

Table  F-7  in  Appendix  F  lists  the  level  of  change  in 
organizational  effectiveness,  following  the  implementation  of 
an  information  system,  reported  by  each  of  the  45  empirical 
studies  reviewed  here. 

In  a  group  of  studies  that  purport  in  some  or  way  other 
to  report  findings  of  an  inquiry  into  the  success  of 
information  system,  only  20%  actually  did  so.  A  serious 
question  arises  as  to  whether  MIS  can  construct  a  formal 
theory  of  system  effectiveness  when  changes  in  effectiveness 
are  not  reported  by  80%  of  its  empirical  efforts?  When  MIS 
claims  to  be  studying  system  effectiveness  yet  has  no  findings 
related  to  it,  MIS  is  in  danger  of  gravitating  into 
irrelevancy  in  the  estimate  of  its  major  constituents:  senior 
line  managers,  MIS  practitioners  and  the  general  public. 
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V.  CONCLUSIONS 

A.  PROBLEMS  IN  ESTABLISHING  CAUSAL  INFERENCE 
1 .  Research  Daaigna 

Much  of  the  concern  about  credible,  appropriate 
research  designs  for  the  examination  of  the  effectiveness  of 
information  systems  has  been  treated  in  the  previous  chapter. 
A  concern  that  still  merits  exercising  is  the  proportion  (two 
thirds)  of  the  studies  in  this  review  that  were  of  the  one- 
shot  case  design. 

According  to  Denzin  (1970),  the  one  shot  approach 
makes  no  attempt  to  approximate  the  virtues  of  experimental 
design:  no  control,  no  control  group,  no  time  dimension. 
Rather,  it  simulates  an  experiment,  not  through  design  but, 
through  analysis,  specifically  multivariate  analysis. 

This  technique  creates  analytic  groupings  out  of  the 
collected  data  after  the  fact,  groupings  that  the  researcher 
should  have  been  seeking  initially.  The  underlying 
vulnerability  of  multivariate  analysis  is  that  any  purported 
causal  relationships  are  tentative  due  to  the  lack  of  a  time 
dimension.  Nagel  concludes  (1961)  that  the  difficulties  of 
basing  causal  inference  on  multivariate  analysis  "are 
notorious . " 

a.  Internal  Validity 

73%  of  the  studies  reviewed  here  were  vulnerable 
to  alternative  explanations  of  their  findings  because  they 
could  not  control  for  the  effects  of  history  and  subject 
maturation  in  their  research  designs.  84s-  were  similarly 
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vulnerable  to  rival  explanations  of  their  findings  because 
they  couldn't  control  for  the  effects  of  selection  bias  and 
subject  mortality  in  their  designs. 

Further,  22%  of  the  studies  were  vulnerable  to  an 
interaction  effect  of  selection  bias  and  subject  maturation. 
Another  4%  of  the  works  had  their  findings  threatened  by  the 
effects  of  instrument  calibration  and  re-test  sophistication. 
Only  four  our  of  the  45  studies  controlled  all  threats  to 
internal  validity:  two  true  experiments  and  two  posttest-only 
control  groups.  91%  of  the  reviewed  studies  were  in  a 
situation  of  having  their  findings  open  to  one  or  more 
alternative  explanations  other  than  those  reported  by  the 
researchers . 

If  this  group  of  empirical  examinations  of  system 
effectiveness  cannot  state,  with  confidence,  that  their 
findings  are  caused  by  the  variables  claimed  by  the 
researchers,  but  might  well  be  caused  by  effects  that  were 
not  controlled  by  the  designs  of  the  studies,  then  they  lack 
a  basis  for  establishing  causal  inference.  They  cannot  be 
the  foundation  upon  which  a  formal  theory  of  information 
system  effectiveness  can  be  built. 

b.  External  Validity 

89%  of  the  studies  reviewed  here  were  vulnerable 
to  the  possibility  that  their  findings  cannot  be  generalized 
to  a  population  beyond  their  sample  subjects  because  of  the 
effect  of  selection  bias  interacting  with  the  experimental 
variable.  Their  vulnerability  to  this  effect  is  inherent  in 
their  research  designs.  11%  faced  the  same  problem  due  to 
the  effect  of  re-test  sophistication  interacting  with  the 
experimental  variable.  Only  4%  of  these  studies  were,  by 
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virtue  of  their  research  designs,  able  to  control  for  any  of 
the  threats  to  external  validity.  Out  of  the  45  studies 
reviewed,  all  were  vulnerable  to  at  least  one  of  the  threats 
to  external  validity. 

While  external  validity  does  not  directly  affect 
the  establishment  of  causal  inference,  it  does  raise  the 
question  of  the  relevancy  of  these  research  efforts  if  they 
do  not  reflect  any  segment  of  the  real  world  beyond  their  own 
samples.  Of  how  much  value  in  the  construction  of  a  formal 
theory  of  system  effectiveness  will  there  be  in  studies  that 
speak  for  nothing  more  than  their  own  research  arrangements? 

3 .  Time  Dimension 

Out  of  the  45  studies  reviewed  in  this  thesis,  89% 
have  no  way  to  account  for  time  due  to  the  structure  of  their 
research  designs.  In  the  words  of  Denzin  (1968),  studies  of 
this  sort  take  a  "snap  shot"  at  one  point  in  time  and  leave 
it  to  the  respondent  to  weave  a  tale  about  events  preceding 
the  interview  or  questionnaire. 

Causal  inference  requires  that  empirical  findings  be 
able  to  determine  that  a  variable  X  preceded  a  variable  Y  in 
time  in  order  to  support  the  explanation  that  variance  in  X 
caused  subsequent  variance  in  Y.  MIS  assessments  of  system 
effectiveness  lack  this  critical  requirement  for  causal 
inference.  Thereby  they  also  lack  another  foundation  for 
building  a  formal  theory  of  system  effectiveness. 

4 .  Data  Collection  Techniques 

More  than  two-thirds  of  the  reviewed  studies  gathered 
data  on  system  effectiveness  variables  by  asking  direct 
questions  of  human  respondents,  whether  by  sel f-completed 
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questionnaire,  survey  interviews  or  by  phone.  Fewer  than  a 
third  used  methods  other  than  the  direct  asking  of  questions. 

This  renders  the  greatest  part  of  the  MIS  studies  of 
system  effectiveness  vulnerable  to  the  possibility  that  the 
data  they  recorded  reflects  something  c^her  than  the  data  that 
the  researchers  believe  they  collected.  The  direct  asking  of 
questions  is  a  method  that  is  obtrusive  and  socially 
demanding . 

Questionnaires  and  interviews  are  suspected  of 
creating  the  very  data  they  record  rather  than  recording  data 
on  events  or  processes  external  to  the  research  act.  As  with 
internal  validity,  obtrusive  methods  raise  the  issue  as  to 
whether  the  reported  findings  of  such  studies  are  due  to  the 
causes  claimed  by  the  researchers. 

a.  Survey  Research  Problems 

(1)  Penetrating  Language  of  Group  Being  Studied. 
Reliance  on  survey  methods  -  whether  self-administered 
questionnaires  or  personal  interviews  -  raises  a  number  of 
problems.  First  is  that  the  interviewer  or  the  writer  of  the 
questionnaire  must  penetrate  the  language  and  symbols  of  the 
group  being  studied  (Denzin,  1970/ .  Otherwise,  the  interview 
or  the  questionnaire  are  carried  out  in  a  relationship  in 
which  the  researcher  interacts  with  the  respondents  as 
strangers  (Denzin,  1968)  .  This  is  a  process  readily 
appreciated  by  anthropologists  undertaking  to  study  an  alien 
culture.  It  is  a  process  ignored  by  academics  pursuing  the 
study  of  a  group  that  seems  to  be  less  remote.  The  assumption 
by  MIS  academics  that  the  languages  and  meanings  used  by  their 
subjects  are  already  known  is  a  fatal  conceit. 
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(2)  Differential  Status  of  Respondents .  A  second 
problem  -  for  which  there  was  no  evident  concern  in  the 
studies  reviewed  here  -  is  that  different  individuals  within 
a  group  occupy  different  social  positions  in  a  group  itself. 
Consequently,  their  perspectives  on  the  group  and  its  success 
or  failures  will  vary  by  position.  The  apparent  operating 
assumption  in  these  studies  was  that  members  of  a  group  are 
essentially  interchangeable  parts.  Each  represents  the  group 
through  undifferentiated  expressions.  This  has  the  effect  of 
yielding  responses  that  are  actually  quite  different  from  what 
the  researchers  believe  they  are  gathering. 

(3)  Obtrusive  Nature  of  Survey  in  Everyday  Life. 
A  third  problem  concerns  the  interaction  between  the 
questionnaire  or  the  interviewer  and  the  respondent.  The 
operating  assumption  in  the  reviewed  studies  seems  to  be  that 
the  questionnaire  or  the  interview  is  neutral,  eliciting  a 
straightforward  recitation  of  facts  or  of  opinions  not 
influenced  by  the  question-asking  act  itself. 

In  actuality,  the  opposite  is  the  case. 
Respondents  are  anything  but  straightforward  with  question 
askers .  This  involves  socioeconomic,  cultural,  ethnic  or 
organizational  status  differences  between  researchers  and 
subjects.  Male  and  female  differences  is  a  common  source  of 
constrained  reporting  by  respondents  (Denzin,  1970).  So  are 
racial  differences.  These  act  to  create  responses  out  of  the 
interaction  itself  rather  than  collect  data  on  a  phenomenon 
outside  the  interaction  such  as,  say,  systems  effectiveness. 
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(4)  Social  Definition  of  the  Research  Process.  A 
fourth  problem  is  that  the  researchers,  the  topics,  the 
respondents,  the  questions,  the  answers,  the  setting,  and  even 
the  questionnaire  form  are  objects  within  a  research 
interaction.  Their  meaning  is  not  objectively  defined  outside 
of,  or  prior  to,  the  research  act.  Rather,  the  social 
definition  of  these  objects  takes  place  as  a  negotiation 
process  within  the  interaction.  This  is  a  process  about  which 
the  researchers  in  this  review  showed  no  awareness. 

Again,  the  data  collected  may  not  provide  an 
accurate  basis  for  the  findings  claimed  by  the  researchers. 
As  they  are  described  by  Reciting  (1966),  "interviews  and 
questionnaires  intrude  as  a  foreign  element  into  the  social 
setting  they  would  describe,  they  create  as  well  as  measure 
attitudes."  Research  methods,  as  Denzin  (1968)  argues,  are 
not  "sterile  a  theoretical  tools." 

(5)  Role  taking  in  Survey  Responses .  A  fifth 
problem  concerns  the  credibility  of  survey  responses.  As  was 
discussed  in  Chapter  IV,  people  take  roles  with  respect  to 
questions.  They  are  vulnerable  to  Guinea-Pig  effects  and  to 
taking  on  roles  such  as  antagonist,  sympathizer,  helper,  test- 
taker,  informant,  protector,  participant  in  a  scientific 
endeavor  or  spokesperson  for  a  group  (answering  not  for 
themselves  personally  but  for  an  imagined  collectivity) . 

These  roles  put  a  respondent  in  a  frame  of  mind 
other  than  that  of  a  straightforward  provider  of  research 
questions.  The  latter  is  probably  the  very  role  not  taken  up 
by  respondents.  Yet,  straightforward  provider  of  answers  is 
the  only  role  believed  to  be  presented  by  survey  researchers. 
The  result  is  that  the  data  collected  by  surveys  means  are 


59 


something  other  than  what  the  researchers  believe  them  to  be. 
Researchers  analyze  these  data  and  report  them  without  any 
suggestion  that  they  are  affected  by  the  very  process  of 
collecting  the  data. 

(6)  Social  Desirability  Effect.  A  sixth  problem  is 
social  desirability.  This  takes  place  within  the  interaction 
between  a  respondent  and  an  interviewer  or  a  questionnaire 
(where  the  responses  are  not  anonymous  or  not  believed  to  be 
truly  anonymous) .  In  North  America,  and  possibly  Western 
Europe,  a  respondent  gives  answers  that  would  be  socially 
acceptable  by  a  standard  known  to  the  respondent  (usually 
middle  class  norms  if  they  are  of  the  middle  or  lower 
classes)  . 

In  the  United  States,  respondents  usually  wish 
to  present  themselves  as  happy,  mentally  healthy,  educated  and 
prosperous.  While  interview  schedules  and  questionnaires  can 
be  const  meted  to  detect  and  discount  socially  desirable 
responses,  there  was  no  indication  that  this  problem  was  dealt 
with  in  any  of  the  reviewed  studies. 

(7)  Fallacy  of  Reductionism .  A  seventh  problem 
with  the  survey  method  is  that  it  necessarily  gathers  data 
from  individuals.  Missing  are  data  about  social  aggregates 
such  as  organizations,  groups,  teams  or  systems.  In  that 
these  45  reviewed  studies  purport  in  some  way  to  examine  the 
effects  of  information  systems,  it  would  seem  appropriate  that 
the  data  collection  would  focus  on  systems  per  se. 

What  these  researchers,  did  instead,  was  to 
focus  on  individuals,  thereby  indulging  the  fallacy  of 
reductionism:  that  a  functioning  whole  is  the  same  entity  as 
the  sum  of  its  component  units.  This  is,  in  the  view  of 
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Wertheimer  (1950),  a  crucial  problem  in  social  research  in 
which  functional  wholes  have  not  been  defined  as  systems  the 
behavior  of  which  is  not  determined  by  that  of  their 
individual  elements. 

A  metaphor  suggested  by  Mill  (1879)  says  that, 
"the  different  actions  of  a  chemical  compound  will  never  be 
found  to  be  the  sums  of  actions  of  its  separate  parts." 
According  to  Nagel  (1961)  functional  wholes  are  systems  which 
cannot  be  built  up  out  of  elements  by  combining  the  latter. 
A  functional  whole  cannot  be  properly  analyzed  from  an 
additive  point  of  view.  A  whole  cannot  be  inferred  from 
properties  displayed  by  its  constituents  in  isolation  from  the 
whole . 

Coleman  (1958)  complained  that  no  matter  how 
numerous  the  correlations,  how  complex  the  analysis,  survey 
studies  focus  on  individuals.  They  tend  to  be,  in  his  phrase, 
no  more  than  ’’aggregate  psychology."  What  these  studies  miss 
are  the  relationships  between  individuals.  They  show  nothing 
about  the  interactions  between  people. 

(8)  Inability  to  Measure  Behavior .  An  eighth 
problem  with  the  survey  method  is  that  it  is  inherently  unable 
to  measure  behavior.  It  can  only  record  attitudes.  As 
discussed  in  the  previous  chapter,  a  survey  that  asks 
questions  about  behavior  can  only  record  a  person's  attitudes 
about  the  questioned  behavior;  a  survey  cannot  show  the  actual 
behavior.  This  is  because,  as  Denzin  points  out  (1968),  the 
emphasis  on  "respondent  reconstruction"  of  past  events,  the 
survey  is  destined  to  gather  data  only  on  attitudes  and  not 
behavior . 
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Coleman  (1969)  described  this  as  a  shortcut 
taken  by  most  researchers.  The  study  of  behavior  requires 
observation  (watching,  participating  or  experiencing)  or 
recording  of  behavior  (archival  records,  audio  tapes,  video 
tapes,  films).  This  requires  time.  It  requires  equipment. 
It  requires  digging  through  records.  It  requires  theoretical 
sampling  (Denzin,  1970)  of  the  places  where  the  behavior  will 
take  place.  The  convenient  assumption  behind  attitudinal 
research  is  that  there  is  a  connection  between  attitudes  and 
behavior;  to  wit,  attitudes  cause  behaviors. 

While  there  is  evidence  that  behaviors  lead  to 
the  development  of  attitudes,  showing  that  the  causality  runs 
from  attitude  to  behavior  has  been  problematic  (Denzin,  1968)  . 
On  the  contrary,  evidence  has  accumulated  showing  that 
accurate  prediction  of  behavior  from  measures  of  attitudes  is 
not  possible  with  the  methods  now  employed  (Tittle  and  Hill, 

1967)  . 

Studying  behavior  takes  a  greater  commitment  to 
scientific  endeavor  than  is  typically  required  for  a  one-shot 
design,  mail-out  questionnaire.  Therefore,  what  is  observed 
in  these  45  studies,  and  throughout  the  general  practice  of 
research  in  management  topics,  is  a  gravitation  to  expediency: 
the  convenience,  the  speed,  the  low  cost  of  the  survey  method. 

In  summary,  the  survey  fails  to  meet  even  the 
minimal  assumptions  for  an  adequate  methodology  (Denzin, 

1968)  . 
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b .  Questionnaire  Problems 

(1)  Fixity  of  Questions .  In  addition  to  the 
preceding  problems  of  surveys  generally  that  afflict 
questionnaires  as  well  as  interviews,  questionnaire  schedules 
have  additional  problems  peculiar  to  their  usage.  The 
reviewed  studies  that  employed  questionnaires  reported  that 
they  were  of  the  standardized  schedule  type.  That  is, 
respondents  were  presented  with  a  fixed  set  of  questions  and 
a  preconceived,  precoded,  finite  set  of  response 
possibilities . 

The  result  of  this  rigid  approach  was  to 
generate  numbers,  through  the  questionnaire  process,  about 
what  was  on  a  researchers  mind  when  the  questions  were 
written.  It  is  a  process  that  taps  little  of  what  is  on 
respondents'  minds.  The  likelihood  is  that  the  response  to 
the  indifference  of  questionnaire  fixity  is  simply  fabrication 
(Denzin,  1970)  . 

(2)  Construction  of  Scales.  A  second  problem  that 
is  endemic  to  the  questionnaire  is  the  matter  of  constructing 
scales.  The  assumption  behind  the  construction  of  scales  on 
attitudinal  questionnaires,  such  as  Likert-type  instruments, 
is,  as  Denzin  explains  it  (1970),  "that  an  underlying  attitude 
variable  is  being  measured.  The  scores  for  any  respondent  are 
assumed  to  represent  his  position  on  the  latent  dimension  and 
the  items  are  assumed  to  reflect  that  dimension." 

Whether  the  assumption  is  upheld  empirically  is 
problematic  for  all  questionnaires  using  scaled  responses  for 
indications  of  attitudes  or  opinions.  The  difficulty  is  that 
no  single  indicator  can  cover  all  dimensions  of  a  variable 
(Denzin  1970)  .  A  single  indicator  has  only  a  probability 
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relation  to  what  it  is  that  a  researcher  actually  seeks  to 
know . 

(3)  Mail  Out  -  Mail  Back  Problems .  A  problem  that 
plagues  the  mail  out-mail  back  self-completed  questionnaire 
is  that  nothing  is  known  about  the  characteristics  of  the 
people  who  did  not  return  a  questionnaire.  This  leaves  this 
technique  vulnerable  to  the  internal  validity  problem  of  bias 
in  the  selection  of  respondents.  A  face-to-face  encounter 
between  question  asker  and  respondent  can,  and  often  does, 
involve  a  "break  off"  by  a  respondent.  That  is,  a  respondent 
suddenly  declines  to  participate. 

The  question  asker  in  these  situations  at  least 
has  the  possibility  of  gathering  some  indications  of  a 
refuser's  age,  sex,  race  or  other  obvious  characteristics. 
Not  so  the  researcher  using  mail  back  responses.  The  chief 
virtue  of  the  mail  out-mail  back  technique  is  that  is  cheap 
and  easy.  It  is  not  a  means  for  gathering  data  that  will 
contribute  to  the  building  of  formal  theory. 

5.  Definition*  of  System  Effectiveness 

The  definitions  of  system  effectiveness  gleaned  from 
these  45  studies  tell  a  great  deal  about  what  is  taking  place 
in  the  endeavors  of  academic  MIS  to  measure  the  performance 
of  systems.  The  field  has  polarized  into  two  schools  of 
thought:  cost-benefit  analysis  and  user  attitudes. 

Actually,  it  is  less  a  case  of  polarization  than  it  is 
of  the  aggressive  user-attitude  camp  driving  cost-benefit  into 
retreat.  The  user-attitude  enthusiasts  view  do  not  care  if 
their  definition  of  system  effectiveness  as  a  resultant  of 
scientific  endeavor.  For  them  it  is  a  matter  of  ideology. 
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The  user-attitudinalists  characterize  the  use  cost- 
benefit  analysis  as  discredited,  traditional  and  infeasible. 
They  view  user  attitudes  as  modern,  feasible  and  appropriate. 
Their  argument,  to  recall  the  discussion  in  the  previous 
chapter,  is  that  while  data  can  be  gathered  on  costs,  nothing 
can  be  gathered  about  benefits.  Benefits  of  information 
systems  are  so  lacking  in  tangibility  as  to  be  rendered 
useless  as  a  way  to  analyze  effectiveness. 

Cost-benefit  analysis  was  appropriate  for  massive, 
batch-processed  transactions  on  mainframe  systems.  They  are 
part  these  measures  of  the  cost-reduction  efficiency 
mentality.  What  value  can  have,  the  attitudinalists  ask,  when 
we  are  examining  the  effectiveness  of,  say,  a  microcomputer- 
basea  decision  support  system? 

In  the  latter  case,  the  emphasis  is  not  upon 
efficiency  but  upon  effectiveness.  It  is  not  upon  quantity 
and  cost  but  upon  quality.  Therefore,  the  very  advancement 
of  hardware  technology  and  the  evolution  of  software  systems 
have  rendered  cost-benefit  analysis  as  deficient  and 
inappropriate . 

The  only  remaining  issue,  in  the  minds  of  the 
attitudinalists,  is  which  particular  attitudinal 
questionnaire,  and  which  specific  question  items,  are  best 
suited  to  gauge  user  satisfaction.  That,  indeed,  is  the  focus 
of  their  research  endeavors.  Not  the  construction  of  a  formal 
theory  of  system  effectiveness  but  the  fine-tuning  of  the 
ultimate  questionnaire. 

Left  out  of  this  approach  are  any  indicators  of  the 
actual  productivity  that  arises  from  a  system  where  efficiency 
is  the  issue  or  the  actual  effectiveness  of  a  new  system  where 
that  is  the  issue.  Missing  are  measures  of  actual 
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productivity  as  distinct  from  users'  opinions  about 
productivity . 

As  the  discussion  of  this  issue  in  the  last  chapter 
indicated,  there  is  good  reason  to  believe  that  the  conceits 
of  the  attitudinalist s  about  the  superiority  of  their 
formulation  of  system  success  is,  in  reality.  A  fig  leave 
covering  their  methodological  nakedness.  As  graduate 
students,  one  suspects,  they  invested  in  learning  the 
prevailing  but  deficient  survey  method  employing  standardized 
Likert-scaled  questionnaires  in  mail  out  techniques  operating 
under  one-shot  case  designs. 

The  opportunity  cost  of  having  learned  that  approach 
has  been  an  ignorance  of  other  approaches.  The 
attitudinalists  are  not  only  ignorant  of  alternative 
approaches,  they  are  ignorant,  of  the  problems  and  deficiencies 
of  their  own  methods. 

These  suspicions  about,  and  characterizations,  of  the 
user-attitudinalists  would  be  no  more  than  material  for 
empirical  testing  and  academic  contention  were  it  not  for 
their  own  aggression.  Not  content  to  demonstrate  that  their 
way  might  be  better,  they  attack  alternative  concepts  of  the 
effectiveness  of  information  systems.  They  force 
practitioners  of  cost-benefit  analysis  into  a  defensive 
posture . 

At  the  very  least,  they  make  difficult  an  attempt  to 
reconcile  different  views  of  effectiveness.  If  the  concept 
of  efficiency  cannot  be  entered  into  the  dominant  dialogue 
about  system  performance,  it  cannot  be  compared  or  combined 
with  user  attitudes. 
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B . INTELLECTUAL  FOUNDATIONS  OF  HIS 

1.  Failure  to  Establish  Causal  Inference 

The  choice  cf  research  designs,  the  vulnerability  to 
threats  to  internal  validity,  the  lack  of  accounting  for  a 
time  dimension  and  the  choice  of  data  collection  methods  - 
considered  separately  and  taken  together  -  in  the  MIS 
empirical  assessments  of  information  system  effectiveness 
provide  no  grounds  for  establishing  causal  inference. 

2.  Inability  to  Build  Formal  Thaory 

The  inability  to  establish  causal  inference  in  MIS 
studies  of  systems  removes  a  foundation  for  building  a  formal 
theory  of  system  effectiveness.  By  derivation,  it  further 
removes  one  basis  for  constructing  a  formal  theory  of 
information  systems.  Not  to  construct  theory  is  to  miss  the 
point  of  academic  endeavor.  In  the  words  of  Nagel  (1961), 
"the  paramount  aim  of  theoretical  social  science  is  to 
establish  general  laws  which  can  serve  as  instruments  for 
systematic  explanation  and  dependable  prediction." 

3 .  A  Discipline  without  Theory 

MIS  is  an  academic  discipline  that  cannot  construct 
formal  theory  about  its  essential  concerns.  Its  has  put 
itself  in  the  position  of  being  unable  to  adequately  describe, 
explain  or  predict  the  very  matters  that  justify  its 
existence.  It  has  traded  the  pursuit  of  explanation  and 
prediction  for  the  puisuit  of  career  security.  Publ\ cation 
is  pursxied  as  a  means  to  tenure  rather  than  as  a  means  to 
supporting  MIS  practitioners.  A  gulf  will  widen  between  the 
academic  world  of  MIS  and  the  practitioner  world  of  MIS. 
Thereby,  does  academic  MIS  flirt  with  irrelevancy. 
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VI .  RECOMMENDATION 3 


A.  TOR  MANAGERS  AND  COMMANDERS 

1 .  Tha  Cost  of  What  la  Not  Known  About  Information 

Syatama 

The  United  States  Navy  has  hundreds  of  thousands  of 
desk-top  microcomputers.  The  very  number  provokes  a  concern 
as  to  how  efficiently  or  effectively  they  are  being  used,  a 
concern  that  emanates  from  high  levels  in  the  Navy.  Read 
Admiral  Paul  Tobin,  the  (J .  S .  Navy's  chief  of  data  automatior 
systems,  wondered  aloud  recently  how  many  of  the  Navy's 
desktop  machines  were  being  used.  He  was  quoted  in  Federal 
Computer  Week  as  saying,  "Let's  make  sure  we're  using  our 
tools.  In  some  way  the  system  has  gotten  ahead  of  the  users. 
[We]  have  to  figure  out  how  to  do  more  [with  the  basic 
systems]"  (Brewin,  1989). 

Senior  military  officials,  much  like  senior  corporate 
executives,  encounter  difficulties  in  dealing  with  the  world 
of  information  technology.  Senior  officials  often  have  little 
grasp  of  the  technology  of  computers .  They  know  that 
information  systems  can  be  used  to  strategic  advantage  but 
they  are  at  a  loss  to  transform  their  wishes  into  successful 
implementations.  Therefore  some  officials  delay  information 
technology  decisions.  Others  avoid  them  a  1 +-ocrether .  Still 
others  take  the  opposite  tack  and  blindly  provide  a  blank 
check  for  the  purchase  of  information  systems,  especially 
microcomputer-based  systems,  without  a  clear  expectation  of 
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the  return  they  expect,  the  results  they  want  or  how  to 
measure  the  effect  of  what  they  getting. 

Admiral  Tobin  has  summarized  what  most  observers  of 
the  approach  by  the  U.S.  military  to  the  new  information 
technology  already  know:  money  has  been  thrown  at  hardware  and 
software  in  the  past  decade  without  a  sense  of  what  was  wanted 
or  of  wha'c  was  achieved.  That  is  not  the  same  thing  as  saying 
that  the  U.S.  military  has  received  no  return  for  its 
expenditures.  Only  that  (1)  it  is  unknown  what  was  achieved 
and  that  (2)  consequent  achievements  will  have  been  the  result 
of  good  fortune  and  the  dedication  of  field  level  activity 
system-enthusiasts  rather  than  as  the  result  of  planning. 

A  good  deal  of  our  alleged  low  productivity  and  low 
payoff  from  in_ormation  systems,  especially  office  automation, 
is  due  in  large  part  precisely  to  the  generous  but 
unscrutinized  fending  for  microcomputer-based  information 
technology.  Millions  have  been  spent  on  computers, 
peripherals,  supolies,  environments  and  training  without  the 
hard  review  that  would  be  given  to  a  turret  lathe  costing  only- 
thousands.  A  lack  of  scrutiny  before  the  office  automation 
funding  frenzy  has  generated  more  managerial  remorse  than 
office  productivity.  In  hindsight,  managers  and  commanders 
are  asking,  "Where  is  the  computer  pay-off?" 

This  discussion  assumes,  of  course,  that  senior 
officials  truly  seek  to  measure  system  performance  to  find 
their  payoff.  It  would  be  understandable  if  some  officials 
did  not  want  to  discover  the  tru^h  about  the  productivity  of 
office  automation.  After  all,  if  you  have  lavished  funds  on 
desk  top  computing  or  information  centers,  the  last  thing  you 
want  to  do  well  or  accurately  is  a  study  of  exactly  how  much 
product i vity-  was  or  was  not  thereby  enhanced. 
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2 .  Becoming  Informed  Consumers  of  Information  Technology 

In  the  world  of  MIS,  there  exist  a  division  of  labor 
between  practicing  managers  and  MIS  academics.  Managers  need 
academic  MIS  to  develop  scientifically  credible  ways  to 
describe,  explain  and  predict  the  working  of  MIS.  Managers 
can't  do.  It  is  the  job  of  academic  MIS.  However,  it  won't 
get  done  if  managers  indulge  the  inadequate  research  methods 
of  academic  MIS. 

Managers  and  commanders  must  re-acquaint  themselves 
with  the  requirements  for  credible  research.  They  must  be 
science  literate.  They  must  become  informed  consumers  of 
academic  MIS  studies  of  such  matters  as  system  effectiveness. 
They  must  be  able  to  know  when  reported  research  results  are 
scientifically  sound.  They  must  beware  of  the  easy  dismissal 
of  cost  benefit  analysis  and  economic  analysis  by  MIS 
researchers  from  the  user  attitude  school  of  effectiveness. 
Managers  must  be  clear  about  the  distinction  between  an 
effective  system  and  people's  perceptions  of  an  effective 
system . 

Specifically,  managers,  as  consumers  of  research  must 
beware  of  studies  that  purport  to  assess  system  effectiveness 
by  using  attitude  questionnaires  to  gauge  system  outputs  as 
well  as  system  inputs.  Attitude  questionnaires  are 
consistent;  they  exhibit  high  correlation  coefficients  within 
their  own  domains.  The  trouble  is  in  relating  attitudinal 
scores  to  independent,  external  grounding  in  behavior. 

.•here  MIS  researchers  deny  that  data  can  be  gathered 
on  system  benefits  (too  intangible  they  say) ,  it  is  not  only 
possible  to  gather  such  data,  it  is  relatively  easy. 

A  further  burden  that  falls  to  managerial  consumers 
of  MIS  research  is  to  grasp  the  place  of  theory.  Managers 
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must  not  let  their  own  action-oriented,  practical  mindsets 
mislead  them  into  an  impatience  with  the  critical  need  for 
formal  theories  in  HIS.  Managers  and  commanders  must  ask 
whether  the  research  results  they  read  are  part  of  an  effort 
to  build  theories  that  can  describe,  explain  and  predict  MIS 
system  effectiveness.  If  not,  what  are  the  researchers  trying 
to  do?  Are  they  simply  doing  studies  to  gain  academic  tenure? 

If  managers  sense  that  academic  MIS,  as  represented 
by  research  reports  in  academic  MIS  publications,  have  no 
relevance  to  their  work,  they  must  make  that  clear  to  the  MIS 
academics.  Unfortunately,  managerial  MIS  does  not  pay  for 
academic  MIS  directly.  The  income  of  academic  MIS  is  not 
dependent  upon  the  pleasure  of  practicing  MIS.  This  structure 
of  third  party  financing  of  research  allows  MIS  academics  to 
be  distracted  from  the  construction  of  theory  into  the 
expedient  pursuit  of  careers. 

However,  MIS  practitioners  do  provide  academic  MIS 
with  its  legitimation,  its  reason  for  existing.  Legitimation 
is  not  a  strong  control  in  the  short  run.  In  the  long  run, 
without  acceptance  by  practicing  MIS,  academic  MIS  is 
irrelevant . 

This  situation  is  different  in  the  interface  between 
MIS  managers  in  the  defense  community  and  academic  MIS 
researchers  attached  to  defense  educational  institutions. 
There,  the  practitioners  provide  not  only  legitimation  but 
also  research  funding. 

MIS  academics  in  the  defense  community  are  held 
accountable  for  their  research  results.  While  this  control 
possibility  is  not  perfectly  exercised,  its  potential  is  large 
for  discipline  g  academic  MIS  research. 
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B.  TOR  TBS  STUDY  OF  SYSTEM  EFFECTIVENESS 
1 .  Research  Daaigna 

a .  Time  Dimension 

Until  MIS  research  designs  for  studying  system 
effectiveness  account  for  the  time  dimension,  they  will  lack 
any  basis  for  causal  inference  (Vitalari,  1985) .  All  MIS 
research  designs  must  gather  data  at  more  than  a  single  point 
in  time.  The  portion  of  the  MIS  studies  that  have  no  concern 
with  time  in  system  assessment  (now  89%)  must  be  reduced  below 
25%  in  the  next  decade.  Otherwise,  MIS  research  may  fail  to 
establish  scientific  credibility.  MIS  researchers  must 
abandon  cross-sectional  analysis  designs  in  favor  of 
longitudinal  designs. 

b.  Randomised  Comparison  Groups 

As  Nagel  (1961)  noted,  it  is  possible  to  advance 
scientifically  without  meeting  the  full  requirements  of  a  true 
experiment,  yet  some  form  of  controlled  investigation  "appears 
to  be  indispensable."  To  that  end,  MIS  researchers  must  not 
only  employ  comparison  groups  in  research  designs  (which  27% 
of  the  reviewed  studies  did)  but  use  comparison  groups  with 
randomized  assignment  of  respondents. 

Only  by  that  means  can  MIS  research  control  for  the 
threats  to  internal  validity  arising  from  h’ story,  subject 
maturation,  re-test  sophistication,  instrument  calibration, 
statistical  regression,  selection  bias,  respond^^t  mortality 
and  the  interaction  of  selection  bias  and  s'  maturation. 
Only  then  can  MIS  research  build  a  confidence  that  its 
findings  actually  reflect  what  MIS  researchers  intend  instead 
of  being  open  to  rival  explanations  by  effect c  which  were  not 
taken  account  of  in  the  research  designs. 
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c.  Quasi -Experimental  Designs 

While  a  great  deal  can  be  learned  from  laboratory 
experiments,  the  rich  and  fruitful  MIS  research  of  the  future 
will  be  done  in  field  settings.  Most  of  it  already  is  <96% 
of  the  reviewed  studies  were  done  in  field  settings) .  The 
difficulty  is  that  the  experimental  virtues  of  repeated 
measures  with  randomized  controls  are  not  now  being 
approximated . 

The  route  to  gaining  the  richness  of  field  setting 
research  with  the  rigor  of  experimental  controls  is  auasi- 
experimental  research  designs  (Campbell  and  Stanley,  1966) . 
MIS  researchers  must  first  declare  the  experimental  variable 
they  are  examining.  They  must  then  gather  data  before  the 
application  of  an  X  variable  and  again  after.  They  should 
make  longitudinal  (multiple  observations) ,  especially  in  the 
posttest  phase.  This  is  not  a  difficult  requirement.  It  can 
be  accomplished  with  survey  methods  as  well  as  with 
unobtrusive  methods.  The  beauty  of  this  approach  is  that 
information  systems  inherently  produce  data  on  key  variables 
that  a  researcher  into  system  effectiveness  will  want  to 
measure . 

To  compare  organizational  functioning  in  a  manual 
work  mode  with  that  in  a  post-implementation  mode,  researchers 
must  commit  to  digging  through  accounting  and  other 
organizational,  hard-copy  archives.  In  the  post¬ 
implementation  phase,  they  can  then  indulge  the  relative 
luxury  of  having  the  system  being  studied  provide  the  data. 

For  an  example  of  this  technique  see  the  one-group 
pretest/posttest  study  of  a  standardized  contracting  and 
purchasing  system  by  Barclift  and  Linson  (1988). 
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2 .  Triangulation 

Triangulation  is  amply  illustrated  by  the  proverb  of 
the  blind  men  reporting  on  the  characteristics  of  an  elephant. 
Each  man  reported  different  characteristics,  depending  on 
whether  he  was  making  a  tactile  examination  of  the  trunk, 
tusks,  legs  or  tail.  The  situation  is  much  the  same  in  MIS 
research.  Each  researcher,  employing  a  single  method  of  data 
collection,  a  non-experimental  design  and  a  single  theoretical 
position  on  the  nature  of  system  effectiveness  is  little 
better  than  a  blind  man.  Only  conceits  and  psychic 
investments  in  current  methods  keep  MIS  researchers  from 
appreciating  the  limits  of  their  empirical  insights. 

When  the  proverbial  blind  men  can  compare,  contrast 
and  locate  their  findings,  they  open  the  potential  to 
describing  the  illusive  reality  of  the  elephant.  Once  MIS 
researchers  use  multiple  methods  of  data  collection,  multiple 
observation  points  in  time,  multiple  comparison  groups  and 
multiple  forms  of  data  then  they,  too,  will  be  open  to  the 
rounded,  realistic  description  of  the  demonstrated  (rather 
than  the  assumed)  effectiveness  of  information  systems. 

a.  Triangulation  of  Data 

(1)  Triangulation  of  Time.  Of  the  reviewed 
studies,  specification  of  time  was  inadequate.  The  most 
specific  mention  was  the  year  in  which  data  were  collected. 
Beyond  that,  no  attention  was  given  to  the  timing  of  data 
collection.  Triangulation  of  time  would  require  data 
collection  with  these  variations:  different  times  of  year, 
different  times  within  a  month  (a  great  deal  of  information 
system  activity  has  a  monthly  cycle),  different  days  of  a  week 
and  different  times  during  a  day  (this  is  a  potent  but 
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neglected  avenue  of  research  into,  for  example,  the  way  a 
night  shift  crew  uses  a  hospital  information  system  without 
the  support  of  daytime  MIS  staff) . 

Not  all  of  these  variations  may  be  relevant  to 
any  particular  examination  of  an  information  system.  However, 
the  likelihood  is  that  one  of  them  will  profoundly  affect  the 
data  gathered  from  any  one  type  of  information  system. 

(2)  Triangulation  of  Place.  All  human  activity  has 
a  spatial  designation.  Many  of  the  reviewed  studies  noted 
the  organizational  location  of  their  data  collection  that 
often  implied  a  geographical  variation.  Yet,  spatial 
differences  were  not  introduced  consciously  as  a  variable. 
Space  differences  are  attended  to  cross-cultural  comparisons, 
but  cross-cultural  comparisons  have  not  been  a  part  of  system 
effectiveness  inquiries. 

The  temptation  in  MIS  research  is  to  assume 
that  place  is  irrelevant  without  further  consideration.  The 
conceit  is  that  influence  of  information  technology  overcomes 
differences  in  time,  culture  or  place.  But  does  it?  Do 
studies  exist  that  establish  that  spatial  location  can  be 
excluded  as  a  critical  independent  variable?  No.  Instead, 
place  is  conveniently  assumed  away:  information  is 
information . 

Yet,  information  is  an  object  in  our  lives 
subject  to  the  same,  situated  negotiations  among  interacting 
people  as  any  other  object  (Boland,  1985) .  The  influence  of 
spatial  location  is  an  empirical  question;  until  its  effect 
is  deliberately  examined,  it  cannot  be  assumed  away. 
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(3)  Triangulation  of  Level  of  Data.  This  thesis 
has  already  exclaimed  upon  the  need  to  move  MIS  studies  on 
system  effectiveness  away  from  data  collected  from  individuals 
to  data  collected  about  systems  and  organizations.  A 
triangulation  of  data  would  require  that  any  study  of  systems 
performance  include  data  from  three  levels:  (a)  individuals 
(which  MIS  studies  now  gather  in  abundance  and  to  the 
exclusion  of  other  levels  of  data),  (b)  systems  (departments, 
teams,  shifts,  and  true  systems  of  people  who  are  connected 
in  relationships  that  are  built  around  producing  an  output 
through  the  mediation  of  hardware  and  software  of  their 
information  technology  no  matter  what  their  formal 
organizational  identities)  and  (c)  interactions  (the 
encounters,  negotiations,  definitions,  avoidances, 
competitions,  cooperations,  and  coordinations  between 
individuals  or  groups  but  not  characteristics  of  the  people 
or  the  groups  per  se) . 

b.  Trimngulmtxon  of  Thmorima 

To  speak  of  triangulating  theoretical  explanations 
of  system  effectiveness  is  premature  in  a  discipline  that  has 
not  developed  a  formal  theory  nor  is  presently  able  to  do  so. 
Yet,  we  have  emerging,  competing  schools  of  thought  as  to  how 
the  concept  of  system  effectiveness  should  be  defined.  Any 
study  of  system  effectiveness  ought,  therefore,  to  approach 
an  empirical  assessment  with  more  than  one  concept  of 
effectiveness . 

A  system's  performance  must  be  studied  in  terms  of 
user  attitudes.  At  the  same  time  it  must  be  studied  in  terms 
of  costs  and  benefits.  Further,  it  must  be  studied  in  terms 
of  the  economic  value  of  information. 
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And,  finally  (without  exhausting  the  MIS  inventory 
of  concepts  of  system  effectiveness) ,  it  must  be  studied  in 
terms  of  organizational  effectiveness. 

c.  Triangulation  of  Methods 

Survey  researchers  already  employ  a  variant  of 
methodological  triangulation  -  within-method  triangulation 
(Denzin,  1970) .  They  use  various  scales  and  multiple 
standardized  schedules  to  tap  attitudes  they  are  examining. 
The  limit  here  is  that  this  takes  place  within  a  single 
method.  This  leads  researchers  to  congratulate  themselves  on 
the  breadth  of  their  methodological  labors  while  deluding 
themselves  that  variations  on  the  same  method  produce  disti ;Ct 
viewpoints.  According  to  Webb  (1966),  each  method  is 
imprisoned  within  its  own  biases  and  distortions.  Combining 
within-method  variations  is  a  first  step  but  it  cannot  be  the 
only  step. 

What  is  required  of  MIS  research  is  the  combination 
of  methods  to  study  the  same  system  or  the  same  phenomenon. 
Thereby  will  the  strengths  of  one  method  compensate  for  the 
weaknesses  of  another.  The  effectiveness  of  a  information 
system  implementation  can  easily  be  assessed  using  unobtrusive 
measures  as  well  as,  say,  attitudinal  questionnaires  as 
Barclift  and  Linson  (1988)  did. 

The  greatest  benefit  of  between-method 
triangulation  is  countering  rival  explanations  for  reported 
findings  that  arise  from  reactive  arrangements.  The  reactive 
effects  of  questionnaires  and  interviews  on  data  were 
discussed  in  the  previous  two  chapters.  In  an  ideal  strategy, 
MIS  researchers  will  combine  survey  questionnaires,  personal 
interviews,  archival  data,  system-generated  measures. 
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participant  observations  and  phenomenological  techniques 
{Webb,  1966;  Denzin,  1970;  Boland,  1985) . 

3.  Long  Term  Fiold  Studios 

Even  with  a  recommendation  for  accounting  for  a  time 
dimension  and  for  triangulating  for  time  as  a  data  variable, 
the  preceding  discussion  has  been  framed  in  the  assumption  of 
a  relatively  short  term  research  project  measured  from  months 
to  as  long  as  two  years.  Covering  the  implementation  of  an 
information  system,  from  system  proposal  to  final,  post¬ 
training  full  implementation  could  easily  cover  two  to  three 
years.  But  MIS  research  must  commit  to  going  beyond  that. 

Once  the  introduction  of  a  system  has  been  assessed, 
MIS  researchers  must  continue  with  follow-up  studies  through 
the  life  cycle  of  a  system.  Considering  that  the  useful  life 
cycle  of  some  information  systems  is  quite  short  (most  are 
shorter  than  a  decade  and  some  scarcely  last  three  years 
before  succumbing  to  obsolescence),  this  is  not  a  burden. 
Once  entry  has  been  made  to  an  organization  to  study  its 
information  systems,  the  greatest  hurdle  is  past.  Unless  MIS 
researchers  have  a  propensity  to  be  wearisome  to  their  hosts 
and  wear  out  their  welcomes,  they  must  extend  their  research 
relationships  for  a  long  view. 

The  payoff  form  this  is  that  different  things  happen 
to  information  system  in  the  long  term  than  are  ever  imagined 
in  the  short  term.  The  most  potent  revelations  in  social 
research  have  come  from  the  longest  studies:  communities, 
organizations,  cohorts  of  people  over  a  lifetime.  The  same 
will  be  true  with  MIS  topics.  In  their  examination  of  the 
introduction  of  a  productivity-enhancing  system  that  cut 
contact-processing  time  in  a  U.S.  Army  installation,  Linson 
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and  Barclift  (1988)  suggested  that  follow-on  research  was 
needed  to  see  if  the  short  term  efficiency,  in  itself,  would 
provoked  user  activities  to  make  greater  demands  on  the  system 
in  the  years  ahead. 

4 .  Unobtrusive  Measures 

A  theme  that  has  been  exercised  in  the  fourth  and 
fifth  chapters  of  this  thesis,  and  echoed  earlier  in  this 
chapter,  is  the  distorting,  biasing  effects  of  the  survey 
method.  This  leads  to  a  recommendation  for  MIS  research  to 
embrace  the  methodological  point  of  Webb  at  al  (1966) ,  for 
the  use  of  data  collection  techniques  that  are  not  reactive 
-  that  is,  that  do  not  lead  to  role-playing,  fabrication, 
social  desireable  response  and  other  behaviors  that  arise  in 
the  research  interaction. 

This  recommendation  is  for  the  use  of  nonreactive  data 
sources,  specifically  archival  data  sources.  This 
recommendation  is  particularly  appropriate  to  MIS  research 
because  the  production  of  information  is  the  point  of 
information  systems.  Some  of  that  information  reflects  the 
performance  of  the  system  itself.  This  does  not  mean  the 
technological  monitoring  of  hardware  system  in  terms  of 
processor  clock  speed  or  input/output  efficiency.  Rather,  it 
means  the  use  of  organizational  accounting  data  on  costs, 
production  and  outputs. 

An  argument  will  arise  from  the  user-attitudinalist s 
that  the  archival  data  produced  by  information  systems  will 
miss  the  intangible  benefits  of  an  information  system  such  as 
higher  morale.  Once  again  this  reflects  an  investment  in  a 
single  method  rather  than  the  reality  of  available  data. 
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One  approach  to  measuring  morale  is  to  use  attitude 
questionnaires  that  purport  to  assess  it.  But  only  attitudes 
will  be  assessed.  Do  the  attitudes  inform  the  behavior  of 
people?  Does  it  direct  what  they  actually  do?  As  an  example 
of  an  alternative  measure  of  morale,  examine  organization 
records  for  the  classic  workplace-safety-valve  for  people  who 
are  unhappy  with  their  jobs:  sick  leave  (Barclift  and  Linson, 
1966)  .  When  people  are  truly  are  fed  up  with  their  work, 
especially  people  below  the  managerial  or  professional  levels, 
they  avoid  the  workplace  by  staying  home  and  calling  in  sick. 
Or,  they  drive  themselves  into  actual  symptoms  of  illness 
produced  from  anxiety  and  job  stress.  Why  then  must  an 
attitude  questionnaire  by  employed  exclusively  to  assess 
morale  when  a  powerful,  triangulating  behavioral  measure  is 
so  available? 
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Colton,  Kent  W. 

Computer  and  Police:  Pattern  of  Success  and  Failure. 
Sloan  Management  Review,  vol.14,  no. 2,  pp. 75-98. 

Winter  1972-73. 

Inventory  of  problems,  not  a  measure  of  success.  No 
metric  of  effectiveness,  only  impressions. 

There  are  indications  that  computer  use  may  lead  to 
centralization  of  power.  Of  those  interviewed  during  the 
on  site  visits,  96%  felt  that  computer  use  would  lead  to 
better  information  and  grater  control  of  the  police 
department  by  those  at  the  middle  and  top  management 
level . 

It  should  be  remembered  that  the  field  of  computer 
technology  is  still  in  its  infancy.  Perfection  should  not 
be  expected  instantly  in  an  area  so  young  and  rapidly 
changing.  On  the  other  hand,  there  is  a  certain  mystique, 
as  well  as  commercial  force,  surrounding  the  application 
of  computers.  This  has  led  to  high  expectations  and  in 
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some  respects  to  oversell.  As  a  consequence,  there  is  a 
need  for  a  continuing  assessment  of  the  benefits  and 
impact  of  computer  use. 


Cron,  William  L.  and  Marion  G.  Sobol 

The  Relationship  Between  Computerization  and  Performance: 
A  Strategy  for  Maximizing  the  Economic  Benefits  of 
Computerization 

Information  and  Management,  vol.6,  no. 3,  pp. 171-181. 

June  1983. 

Surprisingly  the  immense  growth  in  business  usage  of 
computer  facilities  and  software  in  the  past  decade  has 
not  been  accompanied  by  much  analysis  of  the  influence  of 
computer  ownership  and  compute  usage  on  the  sales,  costs, 
and  profitability  of  the  firm.  Part  of  the  problem  is  one 
of  cause  and  effect:  whether  higher  computer  usage  leads 
to  more  profit  or  whether  higher  levels  of  profit  lead  to 
more  computer  usage.  A  second  difficulty  is  that  of 
estimating  the  sales  or  profits  if  computer  systems  had 
not  been  introduced. 

They  never  discussed  how  performance  should  be  measured 
or  related  to  these  variables. 

Other  attempts  at  evaluating  performance  of  information 
systems  have  relied  on  management  opinions,  based  on  user 
surveys.  Thus  Neumann  and  Segev  state  "a  universal 
measure  like  net  effect  on  earning  cannot  be  applied  yet". 

In  summary,  the  literature  shows  that  relatively  few 
studies  of  these  effects  have  been  made. 

Result  of  this  study  indicate  that  computerization  is 
positively  related  to  the  overall  performance  of  medium 
and  large  size  firms.  Small  firms  should  be  cautious  in 
estimating  the  potential  for  significant  economies  of 
scale  from  computerization,  but  should  use  information  to 
facilitate  management  control  and  decision-making. 


Decker,  Jane  E.  and  John  P.  Plumlee 

Microcomputers,  Manpower  and  Performance  in  a  Public 
Utility 

Public  Productivity  Review,  vol.9,  no. 2-3,  pp. 213-225. 
Summer/Fall  1985. 

Rarely,  if  ever,  can  managers  simply  deploy  microcomputers 
into  the  organization  and  receive  a  clear-cut  productivity 
bonus . 
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The  Performance  Improvement  Division  sought  and  acquired 
microcomputers  not  to  create  a  system  but  to  enhance  its 
own  productivity  within  that  system. 

The  JEA  experience  demonstrates  dramatically  that  the  new 
technology  of  microcomputers  provides  real  opportunity  for 
the  enhancement  of  public  sector  productivity. 


Durand,  Douglas  E.,  Rex  0.  Bennet  and  Samuel  Betty 

What  Does  Information  technology  "Do"  to  Business 
Communications? :  Two  Empirical  Studies 
Information  and  Management,  vol.13,  no. 3,  pp. 111-117. 
October  1987. 

Yet  surprisingly  little  empirical  research  has  been 
conducted  to  determine  the  effect  of  information 
technology  on  business  communications. 

Productivity  can  be  divided  into  two  segments:  efficiency 
and  effectiveness.  Both  studies  suggest  that  end-user 
word  processing  communications  result  in  different  output 
than  those  produced  by  conventional  means. 
Organizationally,  productivity  will  decline  unless  the 
communication  is  more  effective.  Does  the  write's 
communication  style  effect  the  manager's  confidence  in 
decisions?  Thus,  important  questions  remain  regarding  the 
effectiveness  of  communications  technology. 


Edelman,  Franz. 

Managers,  Computer  System,  and  Productivity 
MIS  Quarterly,  vol.5,  no. 3,  pp.1-19. 

September  1981. 

One  measure  of  system's  effectiveness  is  the  degree  of  its 
organizational  impact. 

The  best  load  to  more  efficient  office  operations  is 
through  more  effective  support  the  managers. 

Each  dollar  invested  at  the  top  of  the  pyramid  in 
effective  system  technology  can  be  expected  to  replace  $X 
(X  dollar)  in  white  collar  labor  cost. 


Ill 


Edstrom,  Anders., 

User  Influence  and  the  Success  of  MIS  Projects:  A 

Contingency  Approach. 

Human  Relations,  vol.30,  no. 7,  pp. 589-607. 

1977. 

There  are  several  reasons  why  one  has  to  resort  perceptual 
data : 

1.  Generally  one  can  estimate  the  cost  for  developing 
a  system  but  not  the  benefits,  mainly  because  a 
large  number  of  these  are  intangible. 

2.  Managers  do  not  seem  to  keep  track  of  these  data  and 
hence  they  are  not  available  for  research  purposes. 

3.  Even  if  we  should  know  the  complete  cost-benefit 
picture,  it  still  would  be  impossible  to  judge  the 
relative  success  with  respect  to  alternative 
solutions  to  the  problem  in  any  objective  sense. 

The  information  needs  of  the  manager  are  thus  crucial  for 
the  design  of  the  system,  and  his  satisfaction  with  the 
outcome  is  certainly  a  very  important  criterion  of  the 
success  of  the  system. 

The  perceived  success  of  an  information  system  is, 
however,  not  only  possible  criterion  of  success  and  it 
seems  to  be  useful  to  investigate  alternative  aspects  of 
the  success. 

The  phase  of  the  development  process  is  a  further 
contingent  factor  and  that  it  is  important  to  distinguish 
between  types  of  actors  in  the  development  process. 


Englander,  Valerie  and  Fred  Englander 

Cost-Effectiveness  of  Computer-Based  Welfare  Detection  in 
New  Jersey 

Public  Productivity  Review,  vol.9,  no. 2-3,  pp. 271-282. 
Summer/Fall  1988. 

The  New  Jersey  program...  entails  a  quarterly  computer 
match  of  the  social  security  numbers  of  welfare 
recipients...  against  the  social  security  numbers  of  those 
on  public  and  private  payrolls. 

Given  that  the  fraud  detection  system  was  implemented 
simultaneously  with  major  national  welfare  reform 
legislation,  the  New  Jersey  program  was  assessed  by 
examining  patterns  in  N.J.  caseloads  and  average  grant 
levels  relative  to  the  national  trends.  This  methods 
generated  an  estimated  savings  of  $45.6  million, 
substantially  greater  than  the  $6.2  million  cost  of 
operating  the  program.  This  is  an  encouraging  result  to 
those  policy  makers  who  have  recently  sought  to  extend 


computer-based  approaches  to  detect  and  prevent  fraud  in 
other  Federal  programs. 


Epstein,  Barry  J.  and  William  R.  King 

An  Experimental  Study  of  the  Value  of  Information 
Omeaa.  vol.10,  no. 3,  pp. 249-258. 

1982. 

Virtually  every  controllable  aspect  of  any  information 
system,  its  design  and  implementation,  are  affected  by  how 
information  is  valued. 

No  satisfactory  measure  has  been  developed  that  has 
general  applicability  and  the  question  of  which  measure 
should  be  applied  in  a  given  circumstance  has  not  been 
adequately  addressed. 

Most  measures  of  the  value  of  information  apply  to  a  unit 
that  is  at  a  low  level  of  aggregation.  If  this  is  the 
sort  of  information  value  concept  that  must  be  used  in 
designing  an  IS,  the  designer  is  left  either  to  design 
systems  idiosyncratically  for  individual,  or  to  aggregate 
information  values  across  individual  in  order  to  construct 
a  single  measure  that  best  reflects  the  organization's 
information  values. 


Franz,  Charles  R.  and  Daniel  Ruby 

Organizational  Context,  User  Involvement,  and  The 

usefulness  of  Information  Systems 

Decision  Sciences,  vol.17,  no. 3,  pp. 329-355. 

Summer  1986. 

Two  factors  reflecting  an  MIS  department's  maturity  (size 
and  age)  were  found  to  reduce  users'  perceptions  of  system 
usefulness  . 

The  role  of  organizational  factors  was  more  complex  than 
first  imagined.  Failure  of  the  second  canonical 
correlation  to  produce  significance  led  to  the  rejection 
of  all  our  hypotheses  about  direct  associations  between 
organizational  context  and  user  involvement. 

Suggested  to  strengthen  the  positive  relationship  between 
involvement  during  design  and  perceived  usefulness. 
Paradoxically,  MIS  departments  with  broader  scopes  seemed 
to  weaken  this  same  relationship. 
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Fuerst,  William  L.  and  Paul  H.  Cheney 

Concepts,  Theory,  and  Techniques:  Factors  Affecting  The 
Perceived  Utilization  of  Computer-Based  DSS  in  the  Oil 
Industry 

Decision  Sciences,  vol.13,  pp. 554-569 
1982. 

Perceived  usage,  for  the  purpose  of  this  study,  was 
defined  as  the  decision  makers'  perception  of  how 
frequently  they  interacted  with  the  DSS  and  how  important 
the  information  obtained  from  the  DSS  was  in  assisting 
them  to  make  decision. 

In  terms  of  general  use,  the  factors  of  user  training 
during  the  implementation  process  and  the  accuracy  of 
output  were  found  to  be  important. 

The  only  factor  of  importance  from  the  characteristics  of 
the  decision  maker  was  the  number  of  years  of  experience 
in  the  present  position,  and  it  was  only  important  for 
specific  DSS  use.  Thus  the  greater  their  years  of 
experience  in  their  position,  the  more  managers  will  use 
the  DSS  for  specific,  personally  initiated  reports. 

T1  3  important  factors  in  the  characteristics  of  the  DSS 
are  accuracy  and  relevancy  of  output. 

Training  and  accuracy  were  found  to  be  important  in 
affecting  general  DSS  use;  and  experience  in  the  present 
position,  user  training,  accuracy  of  output,  and  relevancy 
of  output  were  important  in  affecting  specific  DSS  use. 
Thus,  the  system's  professional  should  stress  those 
factors  when  designing,  building,  and  implementing  a  DSS 
in  order  to  maximize  the  degree  to  which  decision  makers 
use  the  system. 


Gallagher,  Charles  A., 

Perceptions  of  the  Value  of  a  Management  Information 
System. 

Academy  of  Management  Journal,  vol.17,  no.l,  pp. 46-55. 
March  1974. 

Difficulty  measure  benefits. A  dollar  comparison  of  costs 
with  benefits  cannot  be  made. 

This  study  included  one  test  of  a  new  methodology.  The 
result  of  a  single  test  cannot  establish  the  validity  of 
a  methodology.  The  study  results  suggest  that  the 
technique  is  basically  useful  in  this  application,  nut 
better  scales  need  to  be  found. 

The  finding  of  this  study  that  managers  in  certain  upper- 
level  managerial  positions  valued  EAB  reports  more  highly 
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than  other  managers  suggest  the  possibility  of  a  general 
relationship  between  managerial  position  and  the  value  of 
information.  Such  a  general  relationship,  if  it  exists, 
should  vary  with  the  type  of  information  and  the  type  or 
organization  structure. 


Gallupe  Brent  R.  and  Gerardine  DeSanctics 

Computer-Based  Support  for  Group  Problem-Finding:  An 
Experimental  Investigation 
MIS  Quarterly,  pp. 277-296. 

June  1988. 

Two  versions  of  the  experimental  task,  one  higher  in 
difficulty  and  the  other  lower  in  difficulty,  were 
administered  to  GDSS-supported  and  nonsupported  decision¬ 
making  group,  yielding  a  2  x  2  factorial  design. 

Groups  with  the  higher  difficulty  task  were  less  satisfied 
than  those  with  the  lower  difficulty  task;  however,  this 
difference  was  not  significant.  The  anticipated 
interaction  effect  between  GDSS  support  and  task 
difficulty  was  not  observed. 


The  only  meaningful  finding  for  this  variable  was  that 
GDSS  support  resulted  in  a  significantly  lower  level  of 
satisfaction  with  the  decision  making  process. 

As  controlled  experimental  study  the  findings  cannot 
necessarily  be  generalized  to  all  levels  or  types  of  GDSS 
software  or  to  actual  organizational  use  of  GDSS 
technology . 


Hall,  P.  G.,  and  T.  J.  Lincoln 

A  Cost  Effectiveness  Study  of  Production  and  Engineering 
System  within  a  tightly  integrated  Manufacturing 
Environment . 

Management  Datamatics,  vol.5,  no. 6,  pp. 262-274. 

1976. 

SESAME  was  primarily  established  to  evaluate  the  cost 
effectiveness  of  the  computer  systems  at  BAC-CAD.  Any 
cost  effectiveness  study  assumes  a  comparison  of  one 
situation  with  another.  Although  the  most  straightforward 
method  of  comparing  costs  and  benefits  would  have  been  to 
compare  the  current  computer  systems  with  the  cost  of  the 
number  of  people  required  to  do  the  same  job,  it  was  felt 
that  this  would  be  a  meaningless  exercise. 
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These  considerations  suggest  that  the  cost  and  benefits 
of  the  computer  systems  are  best  compared  with  the 
realistic  manual  systems  that  would  probably  be  intioduces 
if  the  computer  was  not  available. 

The  basic  philosophy  of  a  SESAME  study  is  that  information 
for  management  is,  in  our  ever  more  complex  environment, 
rapidly  becoming  one  of  the  most  important  of  company 
resources  and  that  good  management  demands  the  most 
effective  utilization  of  the  information  resource  that  can 
be  professionally  obtained. 


Hogue,  Jack  T.  and  Hugh  J.  Watson 

An  Examination  of  Decision-Makers'  Utilization  of 
Decision  Support  System  Output 

Information  and  Management,  vol.8,  no. 4,  pp. 205-212. 
August  1985. 

Decisions  are  considered  to  include  three  major  phases  - 
intelligence,  design,  and  choice.  In  order  for  a  DSS  to 
support  the  intelligent  phase,  it  must  provide  information 
of  a  descriptive  nature . 

The  primary  justification  for  the  development  of  a  DSS  is 
that  it  will  be  of  value  to  the  decision  maker. 

Monetary  assessment  of  success  is  probably  not  likely 
since  it  is  rare  for  a  DSS  user  to  formally  evaluate  the 
impact  of  the  DSS  on  profits.  It  may  be  that  the  use  of 
a  DSS  constitutes  a  successful  system.  This  study  has 
suggested  some  combination  of  satisfaction,  acceptance, 
and  frequency  of  use  as  potential  indicators  of  success. 


Kapsales,  Peter 

Before  and  After  WP :  An  Office  Automation  Productivity 
Study 

Journal  of  System  Management,  pp.7-9. 

April  1986. 

If  I  buy  $7  0 , 000-worth  of  word  processing  (WP)  equipment, 
my  operators  must  still  type  the  names  and  addresses.  How 
much  productivity  is  gained  by  word  processing,  when  the 
number  of  key  strokes  is  not  reduced? 

Managers  must  decide  what  the  potential  gain  in 
productivity  is,  and  if  it  is  large  enough  to  justify  the 
cost  of  the  new  equipment.  It  is  a  difficult  decision 
when  there  is  not  a  reduction  of  keystrokes. 
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Is  Word  Processing  Productive? 

WP  equipment  is  justified  when  it: 

a.  Saves  the  typist  from  making  unnecessary  keystrokes. 

b.  Speeds  up  and  simplifies  the  document  formatting 
activities . 

c.  Provides  significant  benefits  by  generating  letters 
which  are  personally  addressed  to  customers. 

Automating  an  office  almost  always  provides  productivity 
improvements,  but  careful  analysis  should  be  performed  to 
make  sure  the  benefits  outweigh  the  costs.  The  manual 
environment  and  the  impact  automation  will  have  on  it  must 
be  carefully  evaluated. 


King,  William  R.  and  Barry  J.  Epstein 

Assessing  Information  Value:  An  Experimental  Study 
Decision  Sciences,  vol.14,  no.l,  pp. 34-45. 

January  1983. 

Comparing  opinions  of  total  system  with  a  linear  model  of 
opinions  about  separate  attributes  of  system. 

Big  error:  Calls  a  scale  internal  after  explicitly 
directing  respondents  to  treat  it  as  ordinal. 

This  study  demonstrates  the  practicality  of  using  a 
multiattribute  approach  to  the  evaluation  of  information 
systems.  These  evaluations  were  compared  with  this 
calculated  from  a  multiattribute  linear  value  model.  The 
two  evaluations  were  found  to  be  highly  correlated. 

The  important  notion  of  considering  alternative  system 
designs  is  often  not  operationally  applied  because  of  the 
complexity  and  cost  involved  in  describing  alternative 
systems  that  are  directed  toward  the  same  managerial 
objective.  The  ease  with  which  systems  may  be  described 
in  multiattribute  terms  suggests  that  the  use  of  this 
approach  may  facilitate  the  explicit  consideration  of 
alternative  designs. 


King,  William  R.  and  Jaime  I.  Rodriguez 
MIS  Quarterly,  pp. 43-51. 

September  1978. 

Most  evaluations  of  information  systems  are  provided  only 
in  efficiency-oriented  terms  on  a  post  hoc  basis  by  system 
users . 

Attitudes  and  value  perceptions  are  an  important  and  often 
neglected  aspect  of  MIS  evaluation. 


The  MIS  evaluation  process  is  thus  a  dynamic  one  in  which 
each  systematic  evaluation  effort  can  lead  both  to  better 
information  systems  and  to  improved  evaluation 
methodologies . 


Larcker,  David  F.  and  Parker  V.  Lessig 

Perceived  Usefulness  of  Information:  A  Psychometric 
Examination 

Decision  Sciences,  vol.ll,  pp. 121-134. 

1980. 

Perceived  importance  will  refer  to  the  quality  that  causes 
a  particular  information  set  to  acquire  relevance  to  the 
decision  maker.  If  the  information  items  are  a  necessary 
input  for  tasks  accomplishment,  then  this  quality  will 
tend  to  increase  the  perceived  usefulness  of  the  set. 

Perceived  usableness  will  refer  to  the  information  quality 
that  allows  a  decision  maker  to  utilize  the  set  as  an 
input  for  problem  solution.  If  the  information  is  clearly 
presented  and  readable  or  can  be  easily  transformed,  then 
this  quality  will  tend  to  increase  perceived  usefulness. 

A  number  of  instruments  have  been  employed  to  "measure" 
the  perceived  usefulness  of  information  for  decision 
making,  but  these  techniques  lack  any  data  on  the 
instrument  reliability  and  validity.  Thus  in  general 
sense,  the  previous  techniques  are  inadequate  and  lead  to 
a  lack  of  comparability  in  research  results. 


Lincoln  T., 

Do  Computer  Systems  Really  Pay-off? 

Information  and  Management,  vol.ll,  no.l,  pp. 25-34. 
August  1986. 

Despite  the  wide  use  of  cost-benefit  forecasting  to 
justify  proposed  system  investment,  executives  remain 
skeptical  of  the  level  of  benefits  actually  achieved. 

Such  a  situation  reinforces  suspicion  in  executive's  minds 
that  benefits  from  computer  systems  are  speculative  and 
probably  marginal. 

Post-audits  of  established  systems,  however  do  not  have 
such  an  obvious  basis  for  comparison  and  it  is  essential 
to  decide  what  the  comparitor  will  be  before  undertaking 
a  study. 
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SESAME  studies  however  are  concerned  with  overall  cost- 
ef  fectiveness  and  it  is  usually  possible  for  system 
boundaries  to  be  defined  which  are  meaningful  to 
management  and  allow  fairly  straight-forward  cost 
allocation.  The  objective  is  to  produced  a  cost  profile 
which  is  accepted  by  senior  executives  as  ruolistic  and 
comprehensive . 

Increasingly  we  see  a  reluctance  to  invest  further  until 
concrete  evidence  is  available  that  existing  system 
investments  have  produced  reasonable  financial  returns. 

User  benefits  are  frequently  declared  as  "intangible”  and 
rarely  post-audited. 

All  too  frequently  there  is  a  lack  of  financial  discipline 
applied  to  the  system  development  process:  it  is  often 
unclear  who  is  accountable  for  benefit  achievement. 


Lucas  Jr.,  Henry  C., 

System  Quality,  User  Reactions  and  The  Use  of  Information 
Systems . 

Management  Informatics,  vol.3,  no. 4,  pp. 207-212. 

1974. 

The  findings  of  this  study  suggest  that  before  undertaking 
new  systems,  the  quality  of  existing  systems  as  rated  by 
users  should  be  satisfactory.  It  may  be  necessary  to 
enhance  or  redesign  existing  systems  before  undertaking 
the  development  of  new  ones . 

A  user's  past  experience  is  his  best  predictor  of  the 
benefits  of  a  new  system,  and  successfully  functioning 
systems  are  prerequisite  to  obtaining  user  cooperation  in 
system  design. 

Though  causality  has  not  been  demonstrated,  a  priory 
arguments  and  the  data  support  the  goals  of  developing 
favorable  user  attitudes  and  high  quality  systems  to 
encourage  the  use  of  information  systems. 


Lucas  Jr.,  Henry  C., 

Performance  and  The  Use  of  an  Information  System 
Management  Science,  vol.21,  no. 8,  pp. 908-919. 

April  1975. 

Information  can  be  used  both  to  find  and  solve  problems, 
though  most  current  information  systems  seem  to  be 
oriented  toward  problem  finding.  Possibly  problem  finding 
data  can  be  provided  under  exception  conditions  specified 
by  users . 


One  of  the  most  important  implications  of  the  model  and 
results  is  that  different  personal,  situational  and 
decision  style  variables  appear  to  affect  the  use  of 
systems.  These  findings  argue  for  more  flexible  systems 
to  support  different  user'  needs. 

If  attitudes  are  highly  negative,  on  the  other  hand,  users 
will  minimize  the  contribution  of  the  system  and  question 
the  validity  of  output  reports.  Attitudes  and  perceptions 
of  computer  systems  should  be  influenced  in  turn  by  the 
quality  of  the  system  as  perceived  by  users. 


Lucas  Jr.,  Henry  C. 

Application  and  Implementation  (Unsuccessful 
Implementation:  The  Case  of  A  Computer-Based  Order  Entry 
System) . 

Decision  Sciences,  vol.21,  no. 8,  pp. 908-919. 

April  1975. 

Dependable  variables  have  usually  focused  on  some  measure 
of  the  level  of  model  or  computer  system  usage.  Where  use 
is  inappropriate,  researchers  have  employed  measures  of 
satisfaction  and/or  favorable  user  attitudes  as  indicates 
of  successful  implementation. 


Lucas  Jr,  Henry  C., 

The  Use  of  Accounting  Information  System,  Action  and 

Organizational  Performance 

The  Accounting  Review,  pp. 735-746. 

October  1975 

In  a  field  study  it  is  difficult  to  obtain  good  measures 
for  operationalized  variables  and  to  control  for 
confounding  variables.  The  lack  of  extremely  strong 
results  can  be  partially  attributed  to  errors  in  measuring 
the  variables  which  were  included  in  the  research. 

Laboratory  experiments  make  it  pcroJLlc  re  control  better 
for  confounding  variables,  particularly  in  measuring 
performance.  In  addition  to  interviews  and 

questionnaires,  the  use  of  protocols  may  be  necessary  to 
develop  an  understanding  of  how  information  is  integrated 
with  the  decision-making  process. 
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Mansour,  Ali  H.  and  Hugh  J.  Watson 

The  Determinants  of  Computer  Based  Information  System 
Performance 

Academy  of  Management  Journal,  vol.23,  no. 3,  pp. 521-533. 
September  1980. 

The  effectiveness  of  the  relationship  between  system 
personal  and  others  in  the  organization  have  an  impact  on 
system  performance.  (They  looked  at  from  Behavioral 
Variable) . 


Mason,  Richard  0., 

Measuring  Information  Output:  A  Communication  Systems 
Approach 

Information  and  Management,  vol.l,  no. 5,  pp. 219-234. 
October  1978. 

Notes,  that  measures  of  behavior  require  direct 
observation.  The  most  frequently  used  measures  of  change 
in  behavior  are  measures  of  ability  to  perform  problem 
solving  tasks  and  tests  of  skills,  although  most  measures 
of  change  in  behavior  are  unique  to  each  situation  and  are 
difficult  to  identify  in  a  complex  situation. 

Productivity  may  be  measured  as  a  ratio  of  useful  output 
to  the  total  number  of  executions  of  a  unit  operation. 

Our  concerns  for  efficiency  require  that  the  output  of 
these  systems  be  related  to  the  input  resources  consumed. 
Only  then  can  managerial  judgments  be  made  as  to  the 
appropriateness  of  the  allocations. 

There  is  an  important  class  of  systems  in  our  society,  for 
which  output  measures  are  more  difficult  to  define.  These 
are  systems  whose  primary  output  is  information. 


Matlin,  Gerald. 

What  is  The  Value  of  Investment  in  Information  System 
MIS  Quarterly,  vol.3,  no. 3,  pp.5-34. 

September  1979. 

Difficulty  in  finding  a  value  is  not  an  appropriate  reason 
for  avoiding  the  subject. 

Are  they  getting  what  they  paid  for  in  MIS? 

What  is  the  value  of  the  company's  investment 
Information  Systems? 


m 


Managers  intuitively  know  that  the  most  of  the  benefits 
of  IS  investments  are  intangible,  difficult  to  measure, 
and  difficult  to  relate  to  profit  result. 

If  the  expenditure  for  the  system  investment  is  approved 
by  appropriate  levels  of  management  and  if  other  valuation 
methods  are  not  appropriate  to  identify  all  benefits,  then 
the  value  of  the  IS  investment  is  equal  to  the  standard 
of  ROI  in  the  company  times  the  cost  of  the  IS  investment. 


Miller,  Jonathan  and  Brenda  A.  Doyle 

Measuring  the  Effectiveness  of  Computer-Based  Information 

System  in  the  Financial  Sector 

MIS  Quarterly,  vol.ll,  no.l,  pp. 107-124. 

March  1987. 

It  was  found  that  performance  varies  widely  between  firms 
and  that  the  better  performers  were  those  where  the  level 
of  performance  matched  the  perceived  importance  of  the 
particular  aspect  of  IS. 

IS  effectiveness  here  is,  at  least  in  part,  a  function  of 
the  relationship  between  perceived  importance  and 
performance  on  individual  information  systems  attributes. 


Money,  Arthur.,  David  Tromp  and  Trevor  Wegner 

The  Quantification  of  Decision  Support  Benefits  Within  the 
Context  of  Value  Analysis 
MIS  Quarterly,  pp. 223-236. 

June  1988. 

A  DSS  that  is  perceived  to  have  value  by  the  potential 
users  is  more  likely  to  be  adopted  and  implemented  than 
one  for  which  no  significant  value  is  perceived. 

The  technique  used,  namely  conjoint  measurement,  is 
considered  a  robust  procedure  for  converting  subjective 
judgments  about  the  relative  importance  of  intangible 
benefits  into  numeric  scores  for  further  analysis. 
Traditional  cost-benefit  approaches  to  evaluating  the 
ef f ect i veness  of  decision  support  systems  are  generally 
regarded  as  inadequate.  More  recent  approaches  account 
for  the  intangible  benefit  analysis,  but  still  attempt  to 
express  benefits  in  terms  of  costs. 

Consequently,  a  more  rigorous  interpretation  of  the 
perceived  benefits  of  a  DSS,  which  are  a  measure  of  value 
of  the  proposed  DSS,  should  greatly  facilitate  the 
decision  on  whether  to  proceed  with  the  next  stage  of 
development  or  acquisition. 
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Neumann..  Seev  and  Eli  Segev 

A  Case  Study  of  User  Evaluation  of  Information 
Characteristics  for  Systems  Improvement 
Information  and  Management,  vol.2,  pp. 271-278. 

1979. 

The  user  of  the  information  and  his  needs  are  the  key  for 
the  system  design.  If  the  system  does  not  fit  the  user's 
needs,  it  will  not  be  used.  Extent  of  use  serves  in  this 
case  as  an  indication  of  the  success  of  the  system. 

Improvement  of  an  existing  system  is  aimed  at  providing 
a  better  response  to  user  needs,  that  is,  increasing  the 
use  of  the  system  by  increasing  user  satisfaction  from 
existing  system,  it  is  necessary  to  first  measure  his 
current  evaluation  of  the  system,  find  out  those  aspects 
of  the  information  which  receive  low  evaluation,  and  then 
improve  them. 

The  findings  of  this  study  indicate  that  the  users  at  the 
bank  had  a  holistic  perception  of  their  information,  and 
that  content  was  a  dominant  characteristic  which  spilled 
over  other  characteristics. 


Poppel,  Harvey  L. 

Who  needs  the  office  of  the  future? 

Harvard  Business  Review,  vol.54,  no. 5,  pp. 146-155. 
November-December  1982. 

Almost  everyone  could  benefit  from  the  new  technology, 
according  to  this  study  of  white-collar  productivity. 

Despite  the  allure,  however,  businesses  are  spending  less 
than  a  nickel  for  such  technology  of  every  dollar  of 
knowledge-worker  salaries  and  internal  support  costs. 

The  fact  is  that  most  decision  makers  are  skeptical  about 
what  managerial  work  stations,  personal  computers,  video- 
conference  rooms,  and  the  other,  newer  icons  of  office 
automation  can  do  for  their  businesses.  These  executives, 
disenchanted  by  their  pervious  exposure  to  ill-conceived 
forays  into  management  information  systems  and  word 
processing,  doubt  tnat  knowledge  workers  will  embrace  the 
new  technology,  and  they  lack  confidence  that  their 
organizations  can  channel  and  measure  the  intended 
benefits.  In  addition,  members  of  the  computer  illiterate 
majority  are  worried  about  whether,  when,  and  how  they 
themselves  will  deal  with  these  new  electronic  tools. 
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Raghunatan,  TS .  and  W.  R.  King 

The  Impact  of  Information  Systems  Planning  on  the 
Organization 

Omega,  vol.16,  no. 2,  pp. 85-93. 

September  1987. 

"...Attempts  to  relate  planning  to  overall  business 
performance,  measured  in  terms  such  as  return-on- 
investment  (ROI),  are  not  fruitful,  primarily  because  even 
such  an  important  activity  as  strategic  business  planning 
is  only  one  of  many  factors  that  influence  ROI." 

These  studies,  together  with  the  limited  ones  done  in  an 
IS  planning  context,  suggest  that  IS  planning  may  be  most 
fruitfully  evaluated  using  a  surrogate  measure  of  its 
organizational  impact  and  measures  of  the  specific 
elements  that  make  up  the  planning  activity. 


Rockart,  John  F. 

The  Changing  Role  of  the  Information  Systems  Executive:  A 

Critical  Success  Factors  Perspective 

Sloan  Management  Review,  vol.24,  no.l,  pp.3-14. 

Fall  1982. 

Three  of  the  four  critical  success  factors  (communication, 
I/S  human  resources,  and  repositioning)  directly  reflect 
this  evolving  staff-oriented  role  definition.  Operating 
responsibility  for  local  I/S  units  is  increasingly  being 
transferred  to  local  line  managements.  The  top  I/S 
executive  today  is  a  "thinker,  planner,  and  coordinator" 
rather  than  a  direct  "implementor  and  doer." 


Ruby,  Daniel  and  Richard  L.  Zeller 

Factors  affecting  the  Success  and  Failure  of  an  Information 
System  for  Product  Quality 
Interfaces,  vol.8,  no. 3,  pp. 70-75. 

February  1978. 

As  a  research  strategy,  the  comparative  case  study 
reported  here  strikes  a  balance  between  large  sample 
survey  techniques  and  one-shot  case  studies.  It  neither 
possesses  the  severe  limitations  of  these  forms  of 
research  nor  makes  full  use  of  their  advantages. 

Our  case  indicates  the  importance  of  several  factors  in 
MIS  implementation. 

1.  At  the  individual  level,  certain  attitudes  were 
found  to  be  more  important  than  others. 

2.  The  organizational  factors  of  complexity,  formality, 
and  centralization  also  affect  implementation. 

3.  Lack  of  involvement  by  system  developers  is  not 
sufficient  to  ensure  failure  if  the  vital  function 
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4. 


of  explaining  the  system  to  ultimate  users  assumed 
by  some  other  knowledgeable  person  in  the  group. 
Strong  management  support  is  instrumental  to  system 
adoption . 


Ruby,  Daniel. 

User  Attitudes  and  Management  Information  System  Use 
Academy  of  management  Journal,  vol.22,  no. 3,  pp. 527-538. 
September  1979. 

Overall,  these  results  support  the  established  notion  that 
user  attitudes  (or  perceptions)  are  significant  correlates 
of  system  use.  Attitudes  are  less  powerful  in  predicting 
subjective  assessments  of  perceived  worth,  although  the 
relationships  are  significant.  The  finding  that  attitudes 
are  more  strongly  related  to  actual  use  than  they  are  to 
measures  of  perceived  worth  has  important  implications. 

User  concerns  are  critical  to  success  of  MIS,  particularly 
concerns  about  the  impact  of  MIS  on  individual 
performance.  Obviously,  these  concerns  should  be 
addressed  during  implementation,  but  in  some  cases  it  may 
be  too  late  by  then. 

A  system  that  does  not  help  people  perform  their  jobs  is 
not  likely  to  be  received  favorably  in  spite  of  careful 
implementation  efforts.  A  system  that  reduces  rewards  for 
users  is  likely  to  meet  with  disaster.  A  logical  and 
often  recommended  approach  to  systems  design  is  to  involve 
users  in  the  design  effort.  Systems  designers  and 
implementation  teams  would  be  well  advised  to  find  some 
means  of  addressing  these  concerns  during  MIS  development. 


Snitkin,  Sidney  R.  and  William  R.  King 

Determinants  of  the  Effectiveness  of  Personal  Decision 
Support  Systems 

Information  and  Management,  vol.10,  no. 2,  pp. 83-89. 
January  1986. 

The  overall  effectiveness  of  the  system  was  assessed  as 
is  typical  in  such  studies,  using  a  perceptual  measure. 
Overall,  this  study  presents  an  interesting  perspective 
on  the  evolving  notion  of  a  "personal  DSS . "  The  evidence 
that  is  analyzed  here  suggest  that  the  pragmatic  notion 
of  personal  DSSs  has  meaning  and  that  some  determinants 
of  the  effectiveness  of  such  systems  can  be  identified. 
While  such  systems  do  not  meet  the  formal  criteria  for 
DSSs  as  they  were  initially  conceived,  they  are  important 
to  the  everyday  functioning  of  business  organizations,  and 
as  such. 
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Srinivasan,  Ananth, . 

Alternative  Measures  of  System  Effectiveness:  Associations 
and  Implications 

MIS  Quarterly,  vol.9,  no. 3,  pp. 243-253. 

September  1985. 

While  acknowledging  the  importance  of  economic  analysis 
of  MIS  value,  researchers  responds  to  the  shifting 
emphasis  from  efficiency  to  user  effectiveness  by  focusing 
either  on  MIS  usage  or  user  perceived  effectiveness. 

From  a  practitioners  perspective,  it  is  important  to 
understand  what  exactly  is  being  measured  when  a  system 
effectiveness  study  is  initiated  in  an  organization. 


Sullivan,  Robert  S.  and  Stephen  C.  Secrest 

A  Simple  Optimization  DSS  for  Productivity  Planning  at 
Dairyman's  Cooperative  Creamery  Association 
Interfaces,  vol.15,  no. 5,  pp. 46-53. 

September-October  1985. 

The  milk  flow  analysis  program,  used  for  daily  production 
planning  and  inventory  forecasting,  relieves  plant 
supervisors  of  about  four  hours  a  day  of  overwhelming  hand 
calculations . 

In  addition,  the  system  has  improved  interaction  between 
all  levels  of  management. 

The  managers  of  milk  processing  plants  in  the  United 
States  rarely  use  computer-based  production  planning  and 
decision  support  system. 

The  benefits  from  using  MFAP  are  obvious  but  somewhat 
difficult  to  quantify  (Then  they  aren't  obvious).  These 
benefits  are  the  result  of  improved  scheduling  on  the 
evaporator,  the  bottle-neck  operation.  MFAP  has  increased 
daily  plant  throughput  by  an  estimated  150,000  pounds  per 
day . 


Swanson,  Burton  E., 

Management  Information  Systems:  Appreciation  and 

Involvement . 

Management  Sciences,  vol.21,  no. 2,  pp. 178-188. 

October  1974. 

It  seems  reasonable  to  conclude  that  MIS  appreciation 
coproduces  (and  is  coproduced  by)  MIS  involvement. 
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APPENDIX  P 


TABLES  OP  FINDINGS 
Table  r-1:  Research  Design  Types 


Studv 

Year 

1 

Pre 

Test 

2 

Comp 

Grp 

3 

Random 

Assicrn 

Campbell 

4  Stanley 

Design  Type 

1 

Colton 

1973 

One  shot  case  study 

2 

Lucas 

1973 

X 

Static  group  comparison 

3 

Swanson 

1974 

One  shot  case  study 

4 

Gallagher 

1974 

One  shot  case  study 

5 

Lucas 

1975 

One  shot  case  study 

6 

Lucas 

1975 

One  shot  case  study 

7 

Hall  4  Lincoln 

1976 

X 

Static  group  comparison 

8 

Barkin  4  Dickson 

1977 

X 

Static  group  comparison 

9 

Edstrom 

1977 

One  shot  case  study 

10 

King  4  Rodriguez 

1978 

X 

X 

X 

True  experiment 

11 

Robey  4  Zeller 

1978 

X 

Static  group  comparison 

12 

Mason 

1978 

One  shot  case  study 

13 

Lucas 

1978 

X 

X 

X 

True  experiment 

14 

Matlin 

1979 

One  shot  case  study 

15 

Neumann  4  Segev 

1979 

One  shot  case  study 

16 

Robey 

1979 

One  shot  case  study 

17 

Mansour  4  Watson 

1980 

One  shot  case  study 

18 

Ball 

1980 

X 

One  group  pretest/posttest 

19 

Larcker  4  Lessig 

1980 

One  shot  case  study 

20 

Edelman 

1981 

X 

Static  group  comparison 

21 

Altmeyer  4  Bozeman 

1981 

One  shot  case  study 

22 

Rockart 

1982 

One  shot  case  study 

23 

Epstein  4  King 

1982 

X 

Static  group  comparison 

24 

Feurst  4  Cheney 

1982 

One  shot  case  study 

25 

Poppel 

1982 

One  shot  case  study 

26 

Bailey  4  Pearson 

1983 

One  shot  case  study 

27 

Cron  4  Sobol 

1983 

One  shot  case  study 

28 

King  4  Epstein 

1983 

One  shot  case  study 

29 

Bruwer 

1984 

One  shot  case  study 

30 

Decker  4  Plumlee 

1985 

One  shot  case  study 

31 

Englander  4  Englander 

1985 

X 

Time  series 

32 

Srinivasan 

1985 

One  shot  case  study 

33 

Sullivan  4  Secrest 

1985 

X 

One  group  pretest/posttest 

34 

Hogue  4  Watson 

1985 

One  shot  case  study 

35 

Lincoln 

1986 

X 

Static  group  comparison 

36 

Franz  4  Robey 

1986 

One  shot  case  study 

37 

Kapsales 

1986 

One  shot  case  study 

38 

Snitkin  4  King 

1986 

One  shot  case  study 

39 

Aldag  4  Power 

1986 

X 

X 

Posttest  only  with  control 

group 

40 

Miller  4  Doyle 

1987 

One  shot  case  study 

41 

Durand  et  al 

1987 

X 

X 

Posttest  only  with  control 

group 

42 

Raghunathan 

1988 

One  shot  case  study 

43 

Baroudi  4  Orlikowski 

1988 

One  shot  case  study 

44 

Money  et  al 

1988 

One  shot  case  study 

45 

Gallupe  et  al 

1988 

X 

Static  group  comparison 
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Table  r-2:  Thraats  to  Internal  Validity 


sal 


STUDY 

YEAR 

His 

Mat 

Tst 

Ins 

Rac 

I  Sal 

Mor 

Mat 

1  Colton 

1973 

-1 

-1 

-1 

-1 

2  Lucas 

1974 

+1 

? 

+1 

+1 

-1 

-1 

-1 

3  Swanson 

1974 

-1 

-1 

-1 

-1 

4  Gallaghar 

1974 

-1 

-1 

-1 

-1 

5  Lucas 

1975 

-1 

-1 

-1 

-1 

6  Lucas 

1975 

-1 

-1 

-1 

-1 

7  Hall  £  Lincoln 

1976 

+1 

? 

+1 

+1 

+1 

-1 

-1 

-1 

8  Barkin  £  Dickson 

1977 

+1 

7 

+1 

+1 

-1 

-1 

-1 

9  Edstrom 

1977 

-1 

-1 

-1 

-1 

10  King  £  Rodriguaz 

1978 

+  1 

+1 

+1 

+1 

+1 

+1 

+1 

+1 

11  Robay  £  Zallar 

1978 

+1 

7 

+1 

+1 

-1 

-1 

12  Mason 

1978 

-1 

-1 

-1 

-1 

13  Lucas 

1978 

+1 

+1 

+1 

+1 

+1 

+1 

+1 

14  Matlln 

1979 

-1 

-1 

-1 

-1 

15  Naumann  £  Sagav 

1979 

-1 

-1 

-1 

-1 

16  Robay 

1979 

-1 

-1 

-1 

-1 

17  Mansour  £  Watson 

1980 

-1 

-1 

-1 

-1 

18  Ball 

1980 

-1 

-1 

-1 

-1 

7 

+1 

+1 

-1 

19  Larckar  6  Lassig 

1980 

-1 

-1 

-1 

-1 

20  Edalman 

1981 

+1 

? 

+1 

+1 

+  1 

-1 

-1 

-1 

21  Altmayar  &  Bozaman 

1981 

-1 

-1 

-1 

-1 

22  Rockart 

1982 

-1 

-1 

-1 

-1 

23  Epstain  &  King 

1982 

+1 

7 

+1 

+1 

+  1 

-1 

-1 

-1 

24  Faurst  £  Chanay 

1982 

-1 

-1 

-1 

-1 

25  Poppal 

1982 

-1 

-1 

-1 

-1 

26  Bailay  &  Paarson 

1983 

-1 

-1 

-1 

-1 

27  Cron  6  Sobol 

1983 

-1 

-1 

-1 

-1 

28  King  £  Epstain 

1983 

-1 

-1 

-1 

-1 

29  Bruwar 

1984 

-1 

-1 

-1 

-1 

30  Dackar  £  Plumlaa 

1985 

-1 

-1 

-1 

-1 

31  Englandar  £  Englandar  1985 

-1 

+1 

+1 

7 

+1 

+1 

+1 

+  1 

32  Srinivasan 

1985 

-1 

-1 

-1 

-1 

33  Sullivan  £  Sacraat 

1985 

-1 

-1 

-1 

-1 

7 

+1 

+1 

-1 

34  Hogua  £  Watson 

1985 

-1 

-1 

-1 

-1 

35  Lincoln 

1986 

+  1 

7 

+1 

+1 

+1 

-1 

-1 

-1 

36  Franz  £  Robay 

1986 

-1 

-1 

-1 

-1 

37  Kapsalas 

1986 

-1 

-1 

-1 

-1 

38  Snitkin  £  King 

1986 

-1 

-1 

-1 

-1 

39  Aldag  £  Powar 

1986 

+  1 

+  1 

+1 

+  1 

+  1 

+1 

+1 

+1 

40  Millar  £  Doyla 

1987 

-1 

-1 

-1 

-1 

41  Durand  at  al 

1987 

+  1 

+  1 

+  1 

+  1 

+  1 

+1 

+  1 

+1 

42  Raghunathan 

1988 

-1 

-1 

-1 

-1 

43  Baroudi  £  Orlikowski 

1988 

-1 

-1 

-1 

-1 

44  Monay  at  al 

1988 

-1 

-1 

-1 

-1 

45  Gallupa  at  al 

1988 

+1 

7 

+1 

+1 

+  1 

-1 

-1 

-1 
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Table  F-3:  Thraata  to  External  Validity 


Interactions : 
Test  Salct 


Studv 

Year 

£  X 

£  X 

1 

Colton 

-1 

2 

Luoaa 

1974 

-1 

3 

Swan* on 

1974 

-1 

4 

Gallagher 

1974 

-1 

5 

Luoaa 

1975 

-1 

£ 

Luoaa 

1975 

-1 

7 

Hall  £  Lincoln 

1976 

-1 

8 

Barkin  £  Dickson 

1977 

-1 

9 

Edstrom 

1977 

-1 

10 

King  £  Rodriguez 

1978 

-1 

7 

11 

Robey  £  Zeller 

1978 

-1 

12 

Mason 

1978 

-1 

13 

Lucas 

1978 

-1 

? 

14 

Matlin 

1979 

-1 

15 

Neumann  £  Sagav 

1979 

-1 

16 

Robay 

1979 

-1 

17 

Mansour  £  Watson 

1980 

-1 

18 

Ball 

1980 

-1 

-1 

19 

Larckar  £  Leasig 

1980 

-1 

20 

Edalman 

1981 

-1 

21 

Altmayer  £  Bozeman 

1981 

-1 

22 

Rockart 

1982 

-1 

23 

Epstein  £  King 

1982 

-1 

24 

Feurst  £  Chaney 

1982 

-1 

25 

Poppa 1 

1982 

-1 

26 

Bailey  £  Pearson 

1983 

-1 

27 

Cron  £  Sobol 

1983 

-1 

28 

King  £  Epstein 

1983 

-1 

29 

Bruwer 

1984 

-1 

30 

Decker  £  Plumlee 

1985 

-1 

31 

Englander  £  Englander 

1985 

-1 

7 

32 

Srinivasan 

1985 

-1 

33 

Sullivan  £  Secreat 

1985 

-1 

-1 

34 

Hogue  £  Watson 

1985 

-1 

35 

Lincoln 

1986 

-1 

36 

Franz  £  Robey 

1986 

-1 

37 

Kapsalea 

1986 

-1 

38 

Snitkin  £  King 

1986 

-1 

39 

Aldag  £  Power 

1986 

+  1 

7 

40 

Miller  £  Doyle 

1987 

-1 

41 

Durand  at  al 

1987 

+  1 

7 

42 

Raghunathan 

1988 

-1 

43 

Baroudi  £  Orlikowski 

1988 

-1 

44 

Money  at  al 

1988 

-1 

45 

Gallupe  at  al 

1988 

-1 

Reaetv 

Arrq 


7 


7 


7 


? 

7 


7 

7 


Multiple 

X 

Interf 
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Table  r-4:  Handling  of  Tina  Dissension 


Handled 

Time 


STUDY 

YEAR 

Dimension : 

1  Colton 

1973 

No 

2  Lucas 

1974 

No 

3  Swanson 

1974 

No 

4  Gallagher 

1974 

No 

5  Lucas 

1975 

No 

6  Lucas 

1975 

No 

7  Hall  £  Lincoln 

1976 

No 

8  Barkin  £  Dickson 

1977 

No 

9  Edstrom 

1977 

No 

10  King  £  Rodriguez 

1978 

Yes 

11  Robey  £  Zeller 

1978 

No 

12  Mason 

1978 

No 

13  Lucas 

1978 

Yes 

14  Matlin 

1979 

No 

15  Neumann  £  Segev 

1979 

No 

16  Robey 

1979 

No 

17  Mansour  £  Watson 

1980 

No 

18  Ball 

1980 

Yes 

19  Larcker  £  Lessig 

1980 

No 

20  Edelman 

1981 

No 

21  Altmeyer  £  Bozeman 

1981 

No 

22  Rockart 

1982 

No 

23  Epstein  £  King 

1982 

No 

24  Feurst  £  Cheney 

1982 

No 

25  Poppel 

1982 

No 

26  Bailey  £  Pearson 

1983 

No 

27  Cron  £  Sobol 

1983 

No 

28  King  £  Epstein 

1983 

No 

29  Bruwer 

1984 

No 

30  Decker  £  Plumlee 

1985 

No 

31  Englander  £  Englander 

1985 

Yes 

32  Srinivasan 

1985 

No 

33  Sullivan  £  Secrest 

1985 

Yes 

34  Hogue  £  Watson 

1985 

No 

35  Lincoln 

1986 

No 

36  Franz  £  Robey 

1986 

No 

37  Kapsales 

1986 

No 

38  Snitkin  £  King 

1986 

No 

39  Aldag  £  Power 

1986 

No 

40  Miller  £  Doyle 

1987 

No 

41  Durand  et  al 

1987 

No 

42  Raghunathan 

1988 

No 

43  Baroudi  £  Orlikowski 

1988 

No 

44  Money  et  al 

1988 

No 

45  Gallupe  et  al 

1988 

No 
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Table  r-5:  Data  Collaction  Tachniquaa 


Study 

Year 

1st 

Method 

Used 

2nd 

Method 

Used 

1 

Colton 

1973 

Mailed  Questionnaire 

Interview  Survey 

2 

Luca* 

1974 

Questionnaire 

3 

Swanaon 

1974 

Archival 

Questionnaire 

4 

Gallagher 

1974 

Questionnaire 

Smantic  Differential 

5 

Lucaa 

1975 

Questionnaire 

« 

L’’»ss 

1975 

Mailed  Questionnaire 

Archival 

7 

Hall  £  Lincoln 

1976 

Interview  Survey 

S 

Barkin  £  Dickson 

1977 

Questionnaire 

Simulation 

9 

Edstrom 

1977 

Interview  Survey 

10  King  £  Rodriguez 

1978 

Questionnaire 

11 

Robey  £  Zeller 

1978 

Questionnaire 

12 

Mason 

1978 

Questionnaire? 

13 

Lucas 

197  8 

Mailed  Questionnaire 

Questionnaire 

14 

Mat 1 in 

1979 

Archival 

15 

Neumann  £  Segev 

1979 

Questionnaire 

16 

Robey 

1979 

Archival 

Mailed  Questionnaire 

17 

Mansour  £  Watson 

1980 

Mailed  Questionnaire 

18 

Ball 

1980 

Quest ionnai re 

Archival 

19 

Larcker  £  Lessig 

1980 

Questionnaire 

20 

Edelman 

1981 

Archival 

21 

Altmeyer  £  Bozeman 

1981 

Phone  Interview 

Mailed  Questionnaire 

22 

Rockart 

1982 

Interview  Survey 

23 

Epstein  £  King 

1982 

Questionnaire 

24 

Feurst  £  Cheney 

1982 

Mailed  Questionnaire 

Mailed  Questionnaire 

25 

Poppel 

1982 

Diary 

Questionnaire 

26 

Bailey  £  Pearson 

1983 

Interview  Survey 

Mailed  Questionnaire 

27 

Cron  £  Sobol 

1983 

Questionnaire 

28 

King  £  Epstein 

1983 

Questionnaire 

29 

Bruwer 

1984 

Questionnaire 

30 

Decker  £  Plumlee 

1985 

Archival 

31 

Englander  £  Englander 

1985 

Archival 

32 

Srinivaaan 

1985 

Mailed  Questionnaire 

33 

Sullivan  £  Secrest 

1985 

Archival 

34 

Hogue  £  Watson 

1985 

Interview  Survey 

Questionnaire 

35 

Lincoln 

1986 

Questionnaire 

36 

Franz  £  Robey 

1986 

Questionnaire 

37 

Kapsales 

1986 

Archival 

38 

Snitkin  £  King 

1986 

Mailed  Questionnaire 

39  Aldag  6  Power 

1986 

Laboratory  Experiment 

Questionnaire 

40 

Miller  £  Doyle 

1987 

Mailed  Questionnaire 

41 

Durand  et  al 

1987 

Laboratory  Experiment 

42 

Raghunathan 

1988 

Mailed  Questionnaire 

43 

Baroudi  £  Orlikowski 

1988 

Mailed  Questionnaire 

44 

Money  et  al 

1988 

Questionnaire 

45 

Gallupe  et  al 

1988 

Panel 

Questionnaire 
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Tabl*  F-6 :  Definition*  of  ■ffoctiveness 


Study 


Ymt _ Definition  of  System  Payoff 


1  Colton  1973 

2  Lucas  1974 

3  Swan a on  1974 

4  Gallagher  1974 

5  Lucaa  1975 

6  Lucaa  1975 

7  Hall  4  Lincoln  i9V« 

8  Barkin  4  Oickaon  1977 

9  Edatrom  1977 

10  King  £  Rodriguaz  1978 

11  Robay  4  Zallar  1978 

12  Maaon  1978 

13  Lucaa  1978 

14  Matlin  1979 

15  Neumann  4  Sagav  1979 

16  Robay  1979 

17  Manaour  4  Wataon  1980 

18  Ball  1980 

19  Larckar  4  Laaaig  1980 

20  Edalman  1981 

21  Altmeyar  4  Bozaman  1981 

22  Rockart  1982 

23  Epatain  6  King  1982 

24  Feurat  &  Chanay  1992 

25  Poppa 1  1982 

26  Bailay  4  Paaraon  1963 

27  Cron  4  Sobol  1983 

28  King  4  Epatain  1983 

29  Bruwar  1984 

30  Dackar  4  Plumlaa  1985 

31  Englandar  4  Englandar  1985 

32  Srinivaaan  1985 

33  Sullivan  4  Sacraat  1985 

34  Hogua  4  Wataon  1985 

35  Lincoln  1986 

36  Franz  4  Robey  1986 

37  Kapaalaa  1986 

38  Snitkin  4  King  1986 

39  Aldag  4  Power  1986 

40  Miller  4  Doyle  1987 

41  Durand  at  al  1987 

42  Raghunathan  1988 

43  Baroudi  4  Orlikowaki  1988 

44  Monay  at  al  1988 

45  Gallupa  at  al  1988 


Syatam  uaaga 

Attituda  of  uaera  toward  ayatem 
Syatam  uaaga 

Dollar  value  information 
Syatam  uaaga 
Syatam  uaaga 
coat  aavinga 
Information  uaaga 
User  influanca 

Attituda  of  uaara  toward  ayatem 

Attituda  of  uaara  toward  ayatem 

Problem  aolving  ability 

Syatam  uaaga 

Value/coat  ratio 

Attitude  of  uaara  toward  ayatem 

Syatam  value 

Performance  factors 

Attitude  of  uaera  toward  ayatem 

Information  uaefulnesa 

Coat  avoidance 

Attitude  of  users  toward  system 
Job  factors 

Attituda  of  uaara  toward  syatam 

Syatam  uaaga 

Office  automation 

Attituda  of  users  toward  system 

Return  on  investment 

Information  value 

Attituda  of  users  toward  system 

Not  clear 

Cost  savings 

Attituda  of  users  toward  system 
Profit 

System  usage 
Cost  benefit 
System  usefulness 
Cost  avoidance 

Attitude  of  users  toward  system 

Decision  quality 

Attitude  of  users  toward  system 

Grammar 

Implementation  of  plan 
Attitude  of  users  toward  system 
Attitude  of  users  toward  system 
Decision  quality 
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Table  F-7 :  Reports  of  Change  in  System  Effectiveness 


Studv 

Year 

Raportad 

Chanaa 

1 

Colton 

1973 

Not 

raportad 

2 

Lucas 

1974 

Not 

raportad 

3 

Swanson 

1974 

Not 

raportad 

4 

Gallagher 

1974 

Not 

raportad 

5 

Lucas 

1975 

Not 

raportad 

e 

Lucas 

1975 

Not 

raportad 

7 

Hall  £  Lincoln 

1976 

25% 

c 

Barkin  £  DJ ‘kson 

1977 

Not 

raportad 

9 

Edstrom 

1977 

Not 

raportad 

10  King  £  Rodriguez 

1978 

Not 

raportad 

11 

Robay  £  Zallar 

1978 

Not 

raportad 

12 

Mason 

1978 

? 

13 

Lucas 

1978 

Not 

raportad 

14 

Matlin 

1979 

1.1 

to  3 . 3 

15 

Neumann  &  Segev 

1979 

Not 

raportad 

16 

Robay 

1979 

Not 

raportad 

17 

Mansour  £  Watson 

i*eo 

Not 

raportad 

18 

Ball 

1980 

Not 

raportad 

19 

Larckar  £  Lassig 

1980 

Not 

raportad 

20 

Edelman 

1981 

15  1 

to  20% 

21 

Altmayar  £  Bozaman 

1981 

Not 

raportad 

22 

Rockart 

1982 

Not 

raportad 

23 

Epstain  £  King 

1982 

Not 

raportad 

24 

Feurat  £  Chanay 

1982 

Not 

raportad 

25 

Poppa 1 

1982 

Not 

raportad 

26  Bailey  6  Pearson 

1983 

Not 

raportad 

27 

Cron  £  Sobol 

1983 

? 

28 

King  fi  Epstain 

1983 

Not 

raportad 

29 

Bruwer 

1984 

Not 

raportad 

30 

Decker  £  Plumlaa 

1985 

? 

31 

Englandar  £  Englandar 

1985 

4  9% 

32 

Srinivasan 

1985 

Not 

reported 

33 

Sullivan  £  Sacrast 

1985 

$100, 000 

34 

Hogua  £  Watson 

1985 

Not 

reported 

35 

Lincoln 

1986 

13- 

400% 

36 

Franz  £  Robay 

1986 

Not 

raportad 

37 

Kapsalas 

1986 

102% 

38 

Snitkin  £  King 

1986 

Not 

raportad 

39  Aldag  £  Power 

1986 

No  improvement  following  implementation 

40 

Millar  £  Doyla 

1987 

Not  raportad 

41 

Durand  at  al 

1987 

Not  raportad 

42 

Raghunathan 

1988 

Not  raportad 

43 

Baroudi  £  Orlikowski 

1988 

Not  applicable 

44 

Money  at  al 

1988 

Not  reported 

45 

Gallupe  at  al 

1988 

Group  decision  support  system  supported  bi 
quality  of  decisions 
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